Alright. Welcome, folks, this is the Thursday night LUG meeting.
Of the St. Louis Linux users group. And we have… I have a couple good topics tonight from our old friend Robert Sittick.
The recording has started. So, Stan, intro over to you.
Oh, anybody that really cares about the general order and sequence of our meetings, you can go to the main web page here and.
Bring up the little slideshow. Welcome to the meeting! Yay!
Got a bunch of slides here, I'm not gonna hit on them all too much real quickly, but just to talk about us generally.
Okay. The echo again?
Uh… We…
Yes, Steve.
I didn't understand that. Anyway, uh… We found a special interest group with a St. Louis Unix users group.
Uh, usually it's 8 days after the job. internal meeting, uh, all the meetings are free and open to the public, there's no membership-free fee at all.
Uh, we have… on our webpage, we have a lot of guidance for people, and talk about our presentations, and… We generally have either just your Zoom meetings.
And, uh, there's some handouts available there. And there's gonna be… we start at 6, arrival time, people can test their chat and their microphone, sound, screen sharing, things like that, and at 6.30, thereabouts, we start with introductions, announcements, etc, etc.
And then we get into the presentation. Are you sure, uh… Chairman James Conroy, he's here tonight. Gary usually comes along and does the MC. I'm doing that right now.
Our host, Lee Lambert of Omnitech. He also is our… the, uh, secretary… no, the, uh, treasurer of our group.
I do production here. I'm also vice president of the group. Gary happens to be president of the.
The general meeting, the SLUG meeting, uh. We need somebody to help volunteer during the meetings to monitor the chat.
Just, uh… Robert gets too involved with the slides, he may not be reading the chat.
Going on. Uh, we also appreciate if we… other people besides Carrie could find speakers.
We always need volunteer speakers. We have lots and lots of suggested topics, but no people who do this presentation.
Uh, this meeting tonight, November 20th, the next meeting will be December 18th, is the.
The, uh… every month. Uh, 5 days from now, usually the fourth Tuesday of the month, is the new Linux user meeting. It's for beginners.
It's a beginner workshop. And then there's a Slack that's usually on the first Thursday of each month, there's a, uh… Hardware and configuration Workshop, and it's a hardware, software.
Operating system agnostic kind of thing, and both of those two meetings, the new Linux and the Slack meeting, are.
our free format. Uh, the next Sloop General Meeting will be on the second Wednesday of each month.
Uh… because those should have been under a rock, Microsoft Windows 10, end of life occurred in October.
Um, any other announcements that anybody would like to make?
Floor's open.
Happy Thanksgiving, early.
is the microphone.
Thank you. For those that are interested, there's a free PDF and book available out there called.
Titled Bye Bye Windows 10. It's available from Amazon as paperback or Kindle. Also, the.
PDF is available free on the internet. Uh, there's also a project called Endof10.org.
It's a project that started in Europe, and it highlights groups.
That, uh, that will help people transition from. Windows 10 to 11, or… from Windows 10 to… Linux. Um… These meetings are… are recorded, uh, for the… and they put the… put them up on the…
The, uh… our archives. Anybody here can be recording?
So, uh, you wanna go… Find any of the details, you can go to that and find that PDF and look at all the details I have there.
We have a calendar that's available at www.schlub.org. slash calendar, and if I look real quickly at that.
See if I can get that up. Oh, come on… Come on, machinery.
There you go. The items in red highlighted our sponsored meetings.
That people may be interested in. Um… A lot of things going on. Ah, Gary's here! Yay!
Prevair. Would you like to take over, or… Let me keep going.
Keep going, Sam, keep going.
Okay. Getting back to where I was… Okay, real quickly, uh… We have mailing lists, we have an announce list, there's very limited traffic, usually it's just instructions on how to get to the meetings.
And those instructions are also. Sent to our discussed mailing list.
And people can join up there, and they can express their opinions and have questions and reviews, and discuss.
Technical things. They could even ask system admin type questions. We have an assist admin group that specifically.
for our open support staff, those people that actually do the work.
Uh… So you don't have to go along to that group and ask the sysaden questions.
In general, we have the general meeting and a sponsor meetings are Lucy-affiliated.
The St. Louis Linux users group, this group is a Linux presentation group.
Formal, uh, presentations. The new Linux group is a beginner workshop. It's informal, free format.
the Slack group is a hardware and software configuration workshop.
And it is, uh, also. pre-format, and so forth.
We can now do a call for help. Anybody have something they'd like to talk about?
Ask questions? Tragedy? It breaks something recently, they need some help.
We got a couple of minutes before we start the show.
Pro tip, don't buy a meter that measures how much power your equipment uses.
It will make you cry.
Oh, how are you else you gonna heat your house?
That's what the natural gas boiler is for.
cautious winner, right?
Oh, he's not using here, the… electronic heaters yet.
Okay. Alright.
Um, you had mentioned, uh, uh… Uh, end of life of, uh… Windows 10, uh, also, and then you're on the calendar, and I thought you were going to mention that on the calendar today.
It's the 40th anniversary of MS Windows 1.0.
I never had the dubious honor of seeing 1.0. I saw Windows 2.0.
I might rather use it than 11.
And it was pure garbage.
Yeah, but you can't…
is 11 really that bad?
I had to smack my wife's Windows 11 system
It's.
45 minutes to get it to use the actual hard drive instead of uploading everything to OneDrive and going, oh, you're out of space, you need to upgrade your subscription!
Hmm.
Nope.
isn't OneDrive Google?
Oh…
No, it's Microsoft. You should uninstall OneDrive.
Hmm.
I have purposely been leaving my wife's computer alone, except I'm forced to touch it.
They want to help.
Happy wife and something like that.
Hmm.
It's just gonna get you in trouble.
uninstall it will, um, stop your machine from working.
Because the desktop is out on OneDrive.
I actually made it, um…
Oh, wow.
bring all that stuff back local, and not…
Um…
back it up to OneDrive, and I think it's…
actually local.
Yeah, once you get everything local, I think you can uninstall OneDrive.
I've found that if you use Windows… Pro that is not a problem, because it's not connected to OneDrive.
It's an optional thing.
I'll fight that battle and ask you if I need to undiscuss.
So…
Any other questions or PSAs?
Okay. Um… presenter tonight, Robert, is SITEC.
Uh, how do you want people to… they want… you want people to hold their comments or questions, or… Interrupt you as you go along. How do you want to do your topics?
Oh, man, always. Please interrupt me whenever something, uh, whenever something comes up.
I have a question. Do you number your slides?
Do I number my slides? I don't have slides.
So, it's… it's… It's a markdown document, and uh… I will… you know what?
Okay, do you… okay. You…
I'm gonna post the links to it in the chat.
Oh. Okay, great.
Okay. As far as I'm concerned, any other questions?
If not, please, Robert, uh… Introduce yourself as you click to it, and… Go about your business with the presentation. Thank you.
Cool, so that whoops. There's one link, and let me get the other one real quick, and I will post that in there as well.
They're talking.
All right. So, if anybody wants to follow along, they're more than welcome to.
Dan, Lee, Gary, everybody else, thank you very much for, uh, for being here. Uh, let me…
Let me get my, uh, my video going as well.
All right, there we go. Greetings, everyone, from the Pacific… no, wait, that's the other… that's the other side. Uh, the Desert Southwest.
Um… As Stan mentioned, um, yeah, I used to be out in St. Louis, Missouri, haven't been out there in a while. Did you get out there again sometime soon.
Probably not in the next few months. I know what it's like for the next few months, and uh… yeah, it's cold enough here, I don't need to go out to St. Louis to.
Um, although today it's actually a little bit warmer.
But we got… we got snow in the mountains today, so… Thankfully, the mountains are over there, not here, but it is kind of cool.
Um, yeah, I used to be in St. Louis, used to hang out with you guys, uh… Would be great to come back and actually do a talk in person. Right now, I'm down in Albuquerque, New Mexico.
Um, and I am a data science instructor here at the Community College of Central New Mexico.
See if I can figure this out. And I'm gonna turn off my video, because you don't… you don't need to look at me, that's… you want to look at the screen instead.
Are you, by chance, presenting from your classroom?
So does that mean we get a degree after we watch this?
I am presenting from my classroom.
Absolutely.
Third degree?
Be happy to send you something your way, no worries.
All right, um, unfortunately I can't see what I'm sharing. Am I sharing my screen?
Yes.
Okay, great. So, uh… Yeah, this is me down here, uh… Uh, this is actually me from a few years ago. I used to have much longer hair, um.
Had it back in a ponytail here, although my face is obscuring my ponytail. Um, this is the class that I teach.
Um, all you need to do is just, uh, look for data science at cnm Ingenuity.
Um, these are a few of my former students here. Um, these are all the fun things that, uh, that we cover.
Right, including things like critical thinking, and SQL, and data wrangling, and uh… yeah, machine learning, and all that kind of fun stuff.
So that's what I'm doing now. Um… One of the things I'm really interested in, of course, is, uh, kind of… kind of hits, how would you say, leads back to the days in St. Louis.
I used to volunteer at a few groups out in the area. One of them was Biteworks.
I don't know if they're still around, I hope they are, they did some great work.
Um, and the other group was the, uh, the BDPA, um, up in, uh, up in North St. Louis, uh, up some really bright kids, uh, um.
how would you say, learn about databases, and things like that, so they can perform in, uh.
Uh, in hackathons and other kinds of competitions. So I've always been fascinated by how do we make things.
How do we make materials more accessible to people, especially those who don't have the means to go out and buy their own computers, or simply don't want to mess up the computer that they have.
That's kind of what I'm hearing a little bit earlier about.
you know, hey, I don't want to touch my significant others, my friends, my whatever's laptop, computer, whatever, because, shoot, as soon as I touch it, if it breaks, it's my fault, kind of thing.
Um, and so… Um, that's… that's what I'm also interested in very much of, how can we set things up in virtual environments elsewhere, so that people can try things out, not be afraid to mess things up.
learn new things, um, and if things do blow up, well, it's no big deal, it's just a virtual machine.
So, um… Some of the things I'm going to be referring to, um… Uh, uh… You can find here on LinkedIn Learning, there's a page that I have, and in fact, I'll push this link as well into the chat.
You know, real quick…
I like to collect websites or, uh, library sites from all over the country that have these.
Uh, this is, believe it or not, these are either where students or friends, um, are currently living, or have been living, I should say.
And, uh, obviously here we are in, uh, in Albuquerque.
Uh, so LinkedIn Learning is there. through the library, if you have a library card, literally you can go to these, uh, go to the… library's website, sign in with your library card, and you'll have a.
At least, here in Albuquerque, I haven't been able to try in all these other places, but at least where my students have gone and tried it out, they have been able to get free access to LinkedIn Learning.
LinkedIn Learning used to be known as Lynda.com. It was basically online tutorials and videos on how to do different things.
Um, a few years ago, Microsoft, excuse me, LinkedIn bought them, and then Microsoft bought LinkedIn.
And so, uh, so yes, they're now owned by Microsoft, um, which… Yeah, you can… it's… it's good and bad in many ways. It's kind of good, and it's kind of bad. One of the things that Microsoft also did a few years ago.
is they bought GitHub. Um, and we'll be talking about GitHub in a little… a little bit, some of the things that we can do.
So, if you want to know more about some of the things that, like data science or new programming languages or databases, or, uh.
Any of that kind of stuff. Linkedin Learning, I think, is a really good starting point to get familiar with things.
One of the things that Microsoft does, or I should say that GitHub does, is it offers Codespaces.
And LinkedIn Learning is integrating more and more with Codespaces.
Um, excuse me, with GitHub Codespaces and GitHub, so they'll provide GitHub.
Uh, how would you say the materials, the actual scripts and, and, uh, um… notebooks within GitHub, and then you can run them on, uh, on the GitHub Codespaces.
So, this is a great way to try things out, play with things without you actually having to mess anything up on your computer.
And so, along those same lines, um, I was thinking, well, geez, what other resources are there that provide this kind of service?
And part of the impetus of why I was thinking that is because I wanted to do something similar years ago.
Um, uh, even when I was in St. Louis, it's like, hey, wouldn't it be great if.
You know, we don't have to install things on people's computers.
Peter, we can just say, hey, here you go, go to a website, boom, you can just try this stuff out. It's on somebody else's computer, and if it blows up, not a big deal.
It doesn't mess up your own computer. Um, and uh… I'm happy to say that there's actually several options available to do that today, especially with Linux.
I try to do something equivalent with Windows Desktop. I have not found a solution yet for that, but I wouldn't be surprised if something like that exists in the near future.
But maybe not. Who knows? We'll see what happens. So, these are the… these are just different sites, uh, if… you guys are… if anybody is not, you know, how would you say, see as a library that is in their community.
Uh, that does have a LinkedIn Learning. and you want to edit, please feel free to fork it. This is a GitHub repo, fork this.
Modify it and send me a pull request, and I'll be happy to add it in.
Okay. So, this is the… this is the journey.
Right, uh, so again, the history was, I… like to have something where people can try things out and break things, right? Because one of the big things when you're learning something new is that fear of breaking something. Either.
or laptop, or what have you, um, that's… that's not a fun thing. It's much more… enjoyable to… break somebody else's stuff, um… And not, how would you say, not be catastrophic.
So, there's a few different environments. Uh, this itself, uh, this page.
Uh, is a, uh, is a Jupyter notebook. Um, and I posted a link in the, in the… Zoom chat. I'm gonna fire this up, um… in these different environments.
And the first one is going to be Kaggle notebooks. And so Kaggle is an environment.
Sure it is. Can I go to this environment where you can learn things and try things out. Actually, we go to their main page.
So you can try things out with. Well, the way they market it is AI and ML.
And it is… that is their main focus, is working on artificial intelligence and machine learning and.
Um, you know, other things in that realm. But it doesn't prevent you from working.
How would you say, in other ways, you can learn Python here, you can learn databases here, um… you can learn a Linux commands and show commands and so forth here. Uh, so it's a really nice way to play, um.
How would you say it's a nice sandbox within the play. Normally, on the upper right-hand corner, right, you can register and then sign in.
But it turns out that you actually don't have to.
do, right? Um, if you want… How would you say, to stay anonymous? You… you're more than welcome to. They don't prevent you from doing it, other than they only provide you with about a 15-minute or so, um.
time to which to play things, or play with things.
That's not terrible, it's just more annoying than anything else. If that becomes too annoying and you don't mind, go ahead and register, sign in, and then you have much more time, and you actually get some other features as well, which I'll talk about in a little bit.
But if you don't sign in. Alright, here you go. Um, oh, there you go. Uh, it, uh, your notebook stops after 60 minutes of continuous use.
Uh, this is one that I started up earlier, and it shut down. I just wanted to show this to you.
That's okay. I'm gonna… I'm gonna clear this out. I'm gonna start the button again. And by the way, the way I got here.
Well, that's starting up, since that takes a little bit. If you go to the main Kaggle page.
If you click on any of these buttons up here, and I'm just going to pick on courses.
Uh, you'll get this little plus sign in the upper left-hand corner that says create.
If you click on create and then click on Notebook.
it will then fire up a notebook. And like I said, if you don't have to sign in, you don't have to… you don't have to do any of that, um… In fact, you can create.
As many notebooks as you want this way, at least.
I have not found a limit yet, and I've fired up quite a few of these.
So this is… this is what the notebook looks like.
Once it fires up, once it starts. Um, everything, uh… yeah, I'm good. Got it.
I'm just gonna try this out, I'm just gonna… Actually, I'm going to go back over here and close this guy, click on here, and we're going to try this out. I'm going to try this command, first this print.
Hello, great. Now, by default.
When you… when a notebook fires up in Kaggle.
by default, it runs Python. Right? It's a Python notebook environment, um… Talk about the other kinds of ways to get to it in a little bit, but it's designed to run Python.
Right, that's what it does. So if you type print hello world, this is the Python command, it, uh, it works just fine.
Uh, if you do something like Echo Hello World.
Which would be a shell command, and you run that, notice it doesn't work. It doesn't like that.
Um, we'll see some ways of getting around that.
Um, same thing, uh, with cat. And some things that we can do.
is… nope. Folks have, uh, what they call an escape sequence for Python.
Escape sequences in Shell are usually backslashes, and Python, uh, these, uh, the escape sequence for saying, hey, run the show command is an exclamation point.
And if we do cat Etsy OS released, one of the things we discover.
is that under the hood, we actually are running Ubuntu.
This particular case, it went to 22204, uh, the, uh… The long-term support release, right? So, basically, we've got a Deviant system underneath that we can actually.
Work with. This is a free Linux environment in which to play. Now, it's not a great environment, but it's.
It's not bad. It gets you up to speed, gets you familiar with some commands.
Uh, you can get to try things out. So let's see what we can do, and what are… what are the limitations of.
Of working in this. First of all, one of the things I like to do is have a look at how much disk space do I have.
Um, and uh… Looks like we have a fair amount of disk space to play with.
Approximately 8 exabytes. I don't know how true that is, but that'd be something we're trying out to see if it, uh, if it really is what it says it is.
Um, if not that, um, it looks like some other places we might have a little bit more or less space to play with.
So, definitely some place where we can download files.
Uh, instead of doing the exclamation point, we can do what is called a cell magic. If we do % percent bash.
Everything in that… everything… Every other command that's in that cell will automatically be interpreted by Bash.
So this is a great way to get familiar with some bash code, right? So here's an example. We're going to do, uh, look at the process table, PSFA UX.
I'm gonna move this, uh, extra stuff here on the right. So, this is our process list.
Clearly, we're running at a very limited environment. This doesn't have everything. This is the main process.
Clearly, we're running… we're running Python. Um, but… We can do normal shell commands.
Okay, um… If there was Pearl, let's see, uh, let's see what we can do with Pearl.
What are the capabilities of, uh, of our, uh, of our environment? And indeed, we could, we could run Perl commands in here as well.
Even though the notebook itself is Python. We could actually do some Perl scripting in this environment.
Um… It's often very convenient.
to not have to put in those percent-percent bash, and.
and exclamation points. And so, one of the things that Jupiter allows us to do.
these Jupyter Notebooks allow us to do is we can create aliases.
And this is rather an interesting way to create an alias. You're basically saying, hey, if I see the PS command, what I really want you to do is run the system ps command, right? So this is… What I'm going to type, this is the system command that I want you to do that's equivalent.
And this gets us close. So if we do that.
we now don't have to put exclamation points or anything else. We have EPS at the UX, uh, and if we run that, notice we don't have to put the exclamation point in there. It does think that.
Or Windows… excuse me. Our display size is smaller, so it does truncate the right-hand side, but we can do that.
Um, so… Before, where we would get the Pro command, we can just type.
For all as is, and we get Hello World.
Um, we can now do echo. Without the exclamation point, and we get echo hello world, right? We get the normal system echo in there.
Um, we can therefore see… let's run an ID and see exactly who we are.
Um, oh, we can't do that. The reason why we can't do this is, even though we've come up here and said, hey, can you please alias this.
command, or the system command to this, uh, uh, Python command.
Uh, it doesn't work if there's already a command, a Python function or Python object, I guess, would be the best way to say it, that already exists. So, Python objects take precedence.
And, uh, ID turns out that ID is indeed a function, uh, that is within Python, and so that can't be written. So that does have to be.
Proceeded with an exclamation point. Turns out we're root.
That's kind of nice, you should be able to do some cool things within root.
Um… Let's even curl some things, right? So that's just… do a curl-I.
And unfortunately, that doesn't work. Turns out that the environment that we have within Cagle does not allow us.
For many websites, I don't think it's all websites, but I think it's an overwhelming number of websites, that it doesn't allow you to go through. Basically, I'm guessing they have some kind of.
The firewall rules or DNS, uh, um… Uh, limitations that don't allow you to go to certain websites, or most websites.
I actually believe it's the other way around. They probably have just a whitelist, and everything else is, uh, no.
Right? Um, but we can do some fun things.
Um, for those of you who are C programmers, uh, absolutely indeed, we can write C files, or C programs, right? So here's a simple hello world.
You can see, uh… we can see that we've created it.
There we go, there's a hello world. Um, I'm just gonna add it real quick. I should be able to do this.
Here we go. There's our C program. Can we compile it?
Um, since I didn't alias it, I'm gonna have to put the exclamation point in front of it.
And, uh, looks like that worked. There we go. And, uh, and let's see if we can actually bring it.
programmers, look at that, you now have a C development environment.
Now, the greatest one, but at least something that you can play with and get started with.
Now, there are definitely some, uh, some limitations. Obviously, we've seen this. We don't have a true terminal.
And, uh, here's what I mean by that. Uh, there is something where we can click on, uh, to view.
There we go. We do have a console. But it doesn't work very well. I mean, we can type in commands, like date.
And, uh… Yeah, doesn't like that. I can type echo… Hello, world!
The D, and if we run that, that does.
to work. Why date doesn't work, and why echo Hello World does. I'm only guessing that it's only using.
Right. Uh…
Maybe building shell commands or something, but I'm not sure why. Also, it's not… Great. Yeah.
Try, uh, putting an exclamation in front.
If you're inside your Python and it's…
There you go. So I still have to do exclamation point, yeah.
Or try your alias of…
Like, that may be a new interactive thing that doesn't have some of your aliases?
Let's see if this works. Well, this would… this might work either way, right?
ran Pearl, so it's reading that alias.
row dash H.
Yeah. Well, let's try this one. I don't know what… I don't remember what the commands of GCC, so it doesn't like that, but let's try GCC-H.
And, uh, okay. So yeah, it looks like you have to do the exclamation point.
Some things don't work that well. Clear, of course, does not clear the screen.
Right? Um… LSF Chalet works reasonably well. We do get a listing of stuff. Um… But not… You know, it's good enough if you need to do something quick and dirty.
Perhaps, uh, if you just want to try some things out.
Um, it works reasonably well, but not… not a great environment, especially when you consider some of the other options that, uh, that we'll see later on.
Great. Um… So, Kaggle has some really nice things. I don't… I don't want to knock it for what its intention is, which is basically to work in Python or R.
I think those are the two main languages, uh, to do AI and machine learning, right? So if you want to, uh, um.
participate in competitions? Do you want to share your code? Do you want to get some pre… you know, they provide you with a ton of data sets. You want to work on those datasets, it's fantastic to do that.
Um, lots of things within that ecosystem, lots of things you can do.
But you are limited if you want to. If you need something that's outside the constraints that it provides, right? So… Um, yep, it's very easy to set up, you can just click New Notebook and start coding.
Right? All you need is, literally. What you saw is all you need. You just need a browser, click on the website, and boom, you're in, up and running, right? Um, you get community data, lots of stuff.
Oh, um, one of the things I did want to show you is if you do log in… Um…
Let me see if, like, if I click it here again. Well, if you do log in, and you confirm.
your identity with them, in addition to CPU and RAM, you will also get GPUs, uh, that you can, uh, that you can work with as well. Again, for free, there's no charge to, uh, for using any of this environment.
That said, it is a somewhat restrictive environment. Right? So, yeah, there's limited shell access. Uh, you can't… you cannot SSH in, you cannot SSH out.
Um, it's ephemeral. Uh, the… disk space that we see here that we're using.
Uh, when this virtual machine that's underneath it dies, it goes away. There are ways to save the information, um, but… If you… if you don't log in, you can't save it. You can download the notebooks.
So if you go click on File and click on Download Notebook, anything you do here will be saved. You'll be able to download it.
Great. Um, you cannot change the operating system. It's Ubuntu.
Uh, what was it, 2204, I think is what it was we had up here?
Yeah, 22, so we're a couple years behind. Um, but it works.
Um, so… Good enough to get familiar with stuff, get your feet wet, try things out.
But not great for, uh, for anything long-term. But something to play with.
So let's move on to our next one. That would be Google's collab.
So here's… here's Google's collab. Colab has one of these neat features called Playground Mode, which I've already entered into.
Um, if you… if you click on this. Actually, I've set it up so whenever I click on it, it automatically gets me the playground mode. But if you're not in playground mode.
Uh, there will normally be an option under here that says, uh, enter playground mode. The advantage of that is.
Uh, nothing gets saved to your drive. You can't break anything.
Right, so, um, the downside is, is if you did some really cool stuff, you have to remember.
to save it. Otherwise, if you close the window, uh, you lose it.
Again, great for playing around, great for trying things out, um…
So let's do that. Again, let's try the print hello world.
And, uh, if we do that, yep, again. just like before, we're in a notebook environment.
Again, the default is Python. Uh, if we want to, we can do Echo of Hello World.
That does not work. In order to get this to work again, we're going to have to proceed with an exclamation point, and that works just fine.
Um… Again, we can have a look and see what we've got here. Um, let's do exclamation point cat, it's, uh… Uh, release, same environment, 2204.
Well, we've got a couple cool things that we can do. Uh, let's go ahead and try ID again.
Um, oops. There we go, exclamation point ID.
Uh, again, we are root. Unlike before, though.
We can go outside and do stuff. In particular, we can go outside and clone… whoops… Got that exclamation point.
I can clone… things from Git. So, if I wanted to, I can now play with things within Git. I can get other people's repositories, I can get my own repositories, download them.
Play with them, um… If I… if I set up an SSH key.
I can SSH, I can push this back into GitHub, or whatever my repository is, GitLab, Bitbucket, what have you.
Great. Um, the other really cool thing. is, uh, unlike.
Kaggle… Colab now offers a terminal.
So in the lower left-hand corner, there's a little button that says Terminal.
We now get an actual terminal. We can actually do things in here.
We'll type clear, we do ls-la. This is a full terminal.
Right? So, uh, let's see, the default shell… is Bash, so I'm using Bash.
Alright, so let's see what else we can do things.
Um, well, I just downloaded this GitHub repo. Um, I'm gonna source it.
What's nice about that is I now get all my aliases that I have.
in that. So I've got Git. I don't have good stuff, but I do have… I do have access to works with… with Git. So I can do Git… clones, I can do git push, I can do comments, or commits.
Alright, I can do check-ins, uh, check-outs, all that kind of fun stuff.
As far as space goes. Tea groups. Let's see how much space I have. Uh, not nearly as much space, I got 100GB drive, 39 is being used, so I have 70GB to play with.
Um, not nearly as much as what apparently is on Kaggle, but 70 gigs is nothing to sneeze at. It's a… it's a decent amount of space to play around with.
Colab doesn't have. I'm trying to remember exactly how it works. If you… if you don't use collab… for a while, uh, it will automatically time you out, and then there's a little bit of a grace period if you right away.
reconnect, you won't lose anything. But, uh, if you extend beyond that grace period, anything that you downloaded into that VM is gone.
It's… it blows it away, right? Um… So that's something to keep in mind. Um… There's some fun things that we can do. Uh, there is a, uh, there's an… there's a folder in here.
Kaggle input. I've been playing around with it, I haven't quite figured out exactly what this is.
Um, make it a little bit bigger. Uh, but basically it's mounted as read-only.
Let's see if we can change that. This is a very common thing for sysadmins to do, is to go in and change these kinds of things. Hey, I don't want that to be read-only.
I want you remount that as read-write. And if we do that, and we rep Kaggle, notice we've gone now from read-only.
to rewrite. And so we should be able to do things on it. So, if I go with something like date.
And put it into Kaggle to input. And then update text, um… No complaints there. Gaggle input date.
There it is. So… At the moment, I don't know if that's useful or not, but it's kind of cool that you can.
Remount things, uh, read only or rewrite. Um, I've tried a couple other things with mounting, they seem to work.
Um, but still… still playing with it. What I'm getting at is this, let's say there's something that you want to mount, let's say, I don't know, Microsoft OneDrive, or your Google Drive, or, uh, you know, something else, we have some kind of a fuse file system.
You could possibly use these tools to do that.
I'm gonna skip over this a little bit. Yes, all these work.
Uh, and again, uh, we can compile things. Let's do that real quick, let's, uh, let's create a, uh… Uh, let's compile that, uh… C code, and whoops.
And let's see if that works. And lo and behold, you can run C code. So… Uh, for those of you who, uh, who like to play with C, here you go, here's another environment in which you can try out your C code and play with things.
Um… Before I do the fork bomb, I want to show you a couple other things. You can mount Google Drive.
Over on the left-hand side, there's a little file thing.
Um, the notebook… Oops. Wow, a bunch of stuff ended up in there. Oh yeah, I typed that in earlier. If we click on this little icon here, that will monitor Google Drive.
If I click on that, it says, hey, I need to run this to do it.
If I click on this, I can connect to my Google Drive.
Eventually. There we go. I have to… I have to allow Google.
Colab to connect to my drive. There you go.
Give that a couple seconds to go. There we go. All right.
So, with that is done. Is that… has now mounted my Google Drive.
Onto this drive. folder. So if I wanted to, I can now go through and do a bunch of stuff on Drive, so… And let's a drive, uh, look at drive, look at my drive, look at my Colab Notebooks.
Yep, there they are. Collab Notebooks. Uh, let's see, there's, uh, things for SLUG.
There we go, we've got a bunch of things in there.
Um… Let's just try 3. Nope, I can't do tree, right? But I can do… number of roof. I can do apt-get update.
I have to get installed. I'm just throwing a bunch of stuff out there.
Again, unlike Kaggle. I can actually install new software to try things out. So if I want to actually install a newer version of Python, or I want to install something else, maybe Rust.
maybe go, maybe something else. Um, I can do that straight here from the command line.
And now I've got the tree command, right? So these are all the cool things about, uh, about trees.
Uh, and lastly, uh, I'm gonna actually unmount my drive real quick, just in case, because who knows whatever happens when that does happen.
Um, do-do. trying to… let's, uh, let's dismiss that.
Go back to terminal…
Um, but again, unmounted, so let's unmount it manually.
Unmount Drive. There it is.
No more drive exists. So, great, I can't blow anything up.
So, one of the great things about, uh, about doing things in an environment is that you can create things like a fork bomb.
Now, obviously, you don't want to do this intentionally, and I'm not doing it on Doom.
It's intentionally just for demonstration purposes. In essence, what this will do is this will use up a bunch of resources.
This is a great example of how to destroy your environment very quickly.
Um, there we go. Uh, I do not recommend doing this on your own machine.
As in, I don't know, your Windows machine, your Linux machine, your Macs.
Unix, whatever. Basically, it's going through and just using up a bunch of resources. We can see the RAM going and increasing.
Uh, eventually what will happen is all the researchers will be used up, and Google will basically terminate this instance.
Like, it goes away. This is a great example of what I mean by you've got these.
throwaway environments. Try these things out, do something, make mistakes.
And it's oh well, not a big deal, we can go and create a new one if we wanted to.
Um… Any questions so far?
Yeah, you were saying that at some point. Uh, this will just time out.
Uh, and you said there was a grace period.
Yeah.
Are we talking a grace period in terms of… minutes, hours, days…
Hmm. As long as you're using it… well, my experience has been, as long as you're using CoLab.
It doesn't kick you out. If you, let's say, walk away for lunch.
Um, I want to say that the time matters somewhere about 20 minutes to half an hour.
Um, of no use, in other words, of idle time.
It then disconnects you. Right, so this… the upper left-hand corner, we've got the little check mark.
Um, if that's there, that means we're connected and everything's good.
Uh, if we're not connected, it'll say something like, you know, connect to runtime, or something along those lines.
Right? Um, and when that happens, you… If you're within that grace period.
You can click on it, and it will reconnect you.
Uh, and you won't lose any of the data that you've got on your virtual drive.
If you're outside that grace period, uh, you lose everything. Now, how long is that grace period? I don't know, but it is definitely on the order of minutes.
Not… not much longer. Um, so, for example. there are times where we've done stuff in class.
And, uh, people have been idle, or we've gone to lunch and come back. When we come back, we have to reconnect, and when we reconnect, anything, for example, any data sets that we've downloaded.
they're gone. And that's… You know, it's not a terrible thing, it's just sometimes, oh, gee whiz, I was right in the middle of, you know, doing the next step.
Okay. Alright.
So you have to be mindful of that. So, I'm just gonna kill this.
At some point, it will… it will… die. But that's okay.
It's gonna trash that, and uh, we're gonna get… I'm just gonna get rid of it completely.
All right. So… Yeah, the con… or the pros and cons of this, it's very easy, right? You have… you don't even have to have… Ooh, um… good question. Now I don't remember if it's, uh… if you want to connect to your Google Drive, you definitely need to have a Google account.
Um, I'm trying to remember if you need one for… to launch this, because.
When I just did it, it automatically logged me in, because we're using Chrome, so I'm already logged in. So, something I need to try.
But I think you can run it with. No, I don't… I… that's not true. I think if you actually want to run it, you actually do need to have a Google account.
It's great for learning and exploring. Um, as I just showed you, it's a throwaway environment. If you want to install things, you can use app-Git install.
or app install, or whatever your favorite version of that is.
Um, it also has GPUs and TPUs, uh, available to it.
Uh, they… those definitely do have a timeout. Uh, as in, you only get a certain number of cycles per day, and if you go through those, you have to wait for the next day to, uh, to try again.
Um, and, uh, yeah, it's got a bunch of cool… it's got a, uh, the environment has got a, uh, a vault, so if you've got secrets and things where you… You want to store your API keys, you can automatically connect to the vault.
And, uh, and download, uh, or connect to APIs and work with them with your, uh, with your API keys in a secure environment.
the items that are in your vault are not stored in your notebook.
They're stored in a separate place, they're separate from your notebook. So if you then later on decide to publish your notebook or share your notebook.
No one else has access to your, uh, to your secrets.
Again, the downsides, there's no SSH. That is, uh, you cannot SSH into it.
Um, it's ephemeral. Right, so after a certain amount of time, you will get terminated.
This used to be true. I wrote this a while ago, so pay… it's paid for terminal. I used to have to get it with Colab Pro. Now if you get Colab Pro, you can have 3 or 4 terminals open at the same time.
Um, but on the free version, you only get one terminal. The cool thing about that is, for me, that's okay, because I know how to use screen.
And, uh, I can install screen, and then basically bounce around between different screen sessions, uh, without having to have multiple… you know, I get multiple turn rolls without having to have multiple turn-offs.
So, uh, that's kind of a cool thing to do. Um, also within that, I can then run different services within just that, uh, that, uh, within that confined environment. So I can have one screen.
running a service in another screen use, uh, um… uh, how would you say, utilizing that service.
Or… yeah. Um… It is isolated, right? It's not designed for hosting continuous services. In other words, you can't.
Hope… you can't open it up to, you know.
provide for your own web hosting or anything like that, your own mail service.
So that's CoLab. The last one I want to show you is Codespaces. So this is in GitHub.
So if you click on this link for Codespaces, uh, you'll go here. You have to be logged in in order to use Codespaces.
So, um, I… that's what I did. They have a bunch of predefined environments, or code space environments, so they've got React and Jupyter Notebooks.NET.
Um, I'm gonna use this blank one here. When you click on that, um, I'm over here.
There you go. I'm gonna click on that again.
Restart Scodespaces. Uh, Codespaces also has a thing where if you don't use it for a certain amount of time, i.e. Your idle time.
It will stop the machine, stop the VM, but it will not delete.
kind of environment, uh, in that sense. Uh, if you want to… actually, let me go over here… Uh… I think it's over here. If you click on Settings.
And you scroll down to Codespaces. Uh… There you go. The default idle timeout, I think the default is actually set to 30 minutes.
Uh, you can set it to, I think… I'm just gonna put a big number in there. There you go.
Half… the maximum is 240 minutes, so 2 hours.
excuse me, 4 hours.
Was there a question?
Alright, so I set mine to 60 minutes, so… Um, if it's running for 60 minutes or more.
Uh, it basically then, uh, it times out. What you have is you have a… you have a, uh, a VS Code environment, so this is a little bit different than what we've seen with Kaggle and.
Colab, so instead of being a Jupyter notebook environment.
We have a VS Code environment. You have our left pane. Uh, if we had a README file, it would show up, and then in the lower right, which I've expanded here, uh, you have a terminal.
Um, and I can go like this and make it really big, uh, so everybody can see it.
Um, and this is a full Linux environment, right? So if I type clear, I get that, ls-a, this is… everything's there. I can do my CAD at C.
press release, uh, and here, instead of being Ubuntu 2202.
Uh, it's 24 by 4. And this… you can change if you wanted to.
Um, I'll leave it up to you to read about it in the… about… dev, containers, and modifying them.
Um, I've had… kind of makes the success.
Codespace has been around for about, uh, 2 years at this point, and I think for… I'm just gonna say, open to the public for about a year and a half.
there's still a few… How would you say?
Beta features, if you will, right? It's not a perfect environment, but every day I use it, I see it's getting better and better and better.
So, uh, back to this, some of the things we can do. Ls-la, echo, right, Canada.
And so, uh, Ezio release. I'm gonna do my git clone again.
Uh, if I try to paste, it pops up and asks me, hey, are you sure you allow this?
I'm gonna say, yes, I allow it. Again, I can clone my things in there.
We're going to have all of the wonderful stuff with Git.
Right? Um, so now I've got my, uh, I've got my useful command line.
shortcuts that I like to use. Right? The amount of disk space that I have?
Uh, not nearly as much. I've got a 32GB drive and 20GB are free.
Um, I've got a little bit more off on temp, I've got 40 gigabytes free.
Um, but that's temporary, right? So… That will go away. If you put anything in temp.
temp gets wiped. Very much like turning your computer back on, right?
Startup attempt gets wiped. So, it's bigger space, but keep in mind that it is temporary storage.
Again, let's have a look at our process list.
Um, where's Process 1? There it is. Well, here we notice that we're running Docker.
Um, it turns out that CAT, both Kaggle and Colab, are also running Docker.
But we can't do anything with it. Right? It's just… it is a container.
But all you get is, you know, the normal shell commands. You can't do anything with Docker in and of itself.
stuff. But Codespaces are different. I'll show that in a little bit.
Right? Um… Do we have Pearl?
Oops. Uh… Yes.
We have Perl, and we do have Perl pre-installed, ready to go. Uh, can I do curling?
And the answer is yes, I can. I'm just doing ahead on example.com, but I can get to everything I've tested so far, as far as getting to a website, I can get to a website with Codespaces.
I am not… I don't appear to be blocked on anything.
Um, and again, just for fun, and I just copied this whole thing.
That is one command. So I'm going to create a… create a file.
I'm gonna list it, and compile it, and then run it.
And there we go. So yeah, I've got my C compiler and everything else if I want to play with C. And of course.
if I want, again, if I want to do Rust, or Go, or Java, or anything else.
If it's not already installed, you can install it. You have full rights.
So let's return back to that Docker thing. The difference in this environment.
Compared to the other two that we've seen, is that not only is it a doctored container, like the others, but we can actually run Docker commands.
Right? Uh, this is what is known as… Docker and Docker, I can actually run Docker within a Docker container environment.
And so, even though I've got Ubuntu 2204 installed.
Uh, if I want to… I can actually run… The door. So here it is, pulling down Fedora.
And… I am now running Dora. Oops, it doesn't know what clear is.
Right? I guess it's a very minimal fedora. But, if we do the OS release, I am now running Fedora Linux.
Uh, it's a container image, right, which is probably going to be a very minimal version of Fedora.
But I now have all the… I… I even forgot, it's RPM? Is that still… yeah, it looks like it. Rpm, I should be able to install whatever I want to and do things with.
Um, anything I want. Um, for those of you who are into containers, yes, you can also do Docker.
Post, you can fire up multiple containers, uh, and create an ecosystem, if you will, so, you know, a database with a web server and a, you know, a caching server and blah blah blah.
Great for trying things out, playing with things, and seeing how things work.
Um… yeah, I'll do the fork bombing a little later on.
Um, one thing you can't… you… Let me… let me just rephrase it. One thing you can do is, if you run a web server.
Um, you can make the ports available to outside people.
Now, the only thing you can do is you can only make ports, uh, 80 and 443.
Uh, available to outside users. Uh, to do these things. It is not meant to be a production environment, it's meant to be more of a testing environment, so they make it available to see for other people to test.
Um, don't run your production web server in a code space, but it's great to try things out and play with things, right?
So… Alright, let's try that fork bomb, just for the fun of it. We've been blow this up.
Actually, any questions on this before I… before I blow this up?
Alright, let's have some fun. So again, it fires off a bunch of processes, it's gonna come in a minute and probably say, hey, resources… exceeded, there you go, resource temporarily unavailable, etc, etc, etc. And eventually, again, as, uh, we didn't see it in Colab, because I killed it beforehand, but after a short while, this will basically say.
hey, this thing isn't available anymore, and it's gone.
Right? So… Some advantages of Codespaces, it's really easy to spin up. You do have to have a GitHub account.
Um, if… If you fire it for, uh, if you start up a code space from an existing repository, uh, your code automatically ends up in the repository, so you can.
You can play around with it. You get a full Visual Studio Code IDE that you can play around with.
There are some limitations. It's not quite the same as the desktop.
But, uh, at least for my use cases, it, uh, it does cover pretty much everything.
If you want to customize it some more, you can, uh, you can look at the DevContainers JSON. Um, I've customized some of my other repos with the Dev Container JSON, where it automatically.
download things, set things up, and all kinds of fun stuff to make it customized for the way I like to work. Um, so you can do that kind of stuff.
Um, and yes, you can do web hosting and some testing.
Again, there is a quota, and uh, I don't see where it says here, but basically you get 120 CPU hours free a month. Uh, that's their free tier.
in two CPU increments, right? So the, uh, the, uh… oh, there you go. It killed my code space. Um… 120 CPU hours a month, but, uh, basically, the, the… machines that they offer are in 2, 4, 8, 16.
CPU increments. Um, and so… basically, for the lowest tier, you get 60 CPU hours a month, and so forth, right?
they start charging you. Now, the thing is, is I don't have a credit card on file with them, so I have yet to find… I've yet to hit that limit, by the way.
Um, of 60 hours. Uh, because basically, you only get charged.
as long as the VM is running. As soon as it stops.
You don't get charged. So that's one of the advantages.
Or how would you say, one of the incentives to actually set your idle time to 30 minutes or lower.
Um, it's, you know, I wrote here, it's get focused, um, you don't have to be tied to a GitHub repository, although it's, it's, I would just say, that's kind of its main purpose, if you will, to do stuff.
Um, but… Yeah, it's… it's a nice machine to play around with, right? It is ephemerable, it is ephemeral. The data persists across the sessions.
But the virtual machine will be stop deleted after whatever that timeout is.
That said, uh, if you don't use a virtual machine, in other words, if you keep it powered down for more than, I think, current… well, the default setting is 30 days, if you… if you don't use it for 30 days, it will actually delete the VM as well.
Again, if there's stuff on that VM that you want to keep, be sure to.
One, just touch the VM, like, turn it on and then turn it off, or… take that data and put it someplace else that's a little bit more, um, a little bit more persistent.
Right? Um, and again, it's not designed for production use. And notice, I didn't say anywhere here that you can't SSH into it.
And technically, you can't SSH into it, but you… you can.
There are ways to basically… you can run things like, uh, if you have an outside machine, you can set up a reverse tunnel, and so you can SSH via the reverse tunnel back in. There's also ngrok, and a few other services that allow you to do something.
Similar. There are other services that are close. to free. Um, I just wanted to put it down here to mention, uh, AWS has a free tier.
Uh, you do have to sign up, you do have to provide them with a credit card, but one of their systems.
or virtual machines has a free tier. And, uh, basically, if you start Cloud9 on that.
You basically… don't get charged, right? And that's true with any of the… any EC2 that's in the free, uh, EC2 is their term for a virtual machine. It's… that's true.
For any virtual machine that uses the free tier, right? Cloud9 just has a nice, uh, nice, uh, IDE interface.
SageMaker is simple. Uh, it provides a Jupyter notebook environment that you'll play with and try things out.
Um… I'm not really a big fan of either, simply because if I'm gonna do stuff in AWS, I'm gonna create my own EC2 instance, install my own Jupyter software, and do whatever I want.
Uh, Cloud9 and SageMaker are more along the lines of, hey, I just need to get up and running and try stuff out.
Uh, these others down here I just mentioned, uh.
They're not technically free, but they do have free credits, especially for new users. Digitalocean does, uh, Google Cloud Platform does, as well as Azure.
Although I didn't spell that correctly. Uh, it's… yeah. But they all have that, and I wouldn't be surprised if other cloud providers do the same thing.
So that's kind of a not-so-quick rundown of a free Linux environments in which to play.
experiment, blow things up, try things out. Any questions on that?
Wow, nice.
All right. So… If no questions, um… Let's go on to the next thing.
Which is… actually using them.
But, um, let's see if I can do this.
Boom. Yeah, that's fine. All right.
Am I sharing my screen still? Does it say DuckDB at the top?
Yes, it does.
All right, so let's try this out. Right? Um, one of the cool things is yesterday I went to a conference, uh, actually, my class and I went to a conference, uh, here in Albuquerque. Uh, it was the, uh, we call it the Data Users.
Um… Data Users Conference. Basically, it's a bunch of people in the area that work in economics, in statistics, in, uh… And the legislature, in various government agencies and so forth, and they provide, you know, they either.
provide or consume public data. And, uh, it's basically, hey, how do we use this data? What's good, what's bad? What could be improved, and how do we improve it, and all that kind of fun stuff.
Um, and, uh, of course, with working with a lot of data.
Um, I don't know how many of you have been playing with public data lately, but there are certain areas of the.
federal government and other places where data is disappearing.
So… It's kind of nice to be able to go through and be familiar with what's there and copy it and do other things with it. Of course, it's also kind of fun to, once you've got a copy.
Or even if you don't have a copy, to play with it, to see what's out there.
And one of the new tools that's in the pipeline, new as in new, I think it's a little bit more than a year old at this point.
maybe… maybe 2 years old, uh, is something called DuckDB.
And, uh, the motivation for GutDB. was, uh, SQLite was a really… is a really nice… what's the word I'm looking for?
transaction database, right? It's very lightweight, minimal, um… And if it meets your needs, it's great.
Right, so only got a few data types with it, but you can do all kinds of SQL queries.
And you can display those SQL queries in many different ways. Html, or JSON, or… Um, many other ways, right? Uh, it's got lots of connectors, so… Uh, you can… you can connect to it from Python and Java and C and, you know, probably everything under the sun at this point.
But it's not so great, what SQL's not so great at, or SQLite, I should say, is not so great at, is doing some really robust analytics.
Um, and working with a bunch of different. data type… or data file types, right? Um, and that's kind of where.
That's kind of where DuckDB comes in. So, whereas in… SQLite, if you, let's say, have a CSV file and you want to work with it, you have to.
take that file and do an ETL process, right? You gotta load the data in, and then you gotta save it as a SQLite database.
And then play with it. Um… Same thing with Excel. Uh, you gotta load it, or transform it into some other format, bring it into SQLite, and then save it as a SQLite database.
Gut DB doesn't do that. You can point that DB at many different file types, and it will.
query it natively, right? It creates an in-memory or in-process, I think is the term they like to use.
And it's not designed to be transactional. Which is what SQLite is designed for. Um, it's designed instead to do, um, analytics, right? So, basically, query.
the data, right? Um… It could be embedded, right, you can, um… Because it's very nice and small, and you can.
basically just pointed at data and then query that data. It has lots of cool analytical stuff.
I don't think I'm going to be able to get to that today, um, but at least I want to show you some of.
Other things that you CAN do with it. Right? Um, how am I gonna run this? Well, I'm just… I've posted a link in the chat, but it's here as well. All you have to do is click on this Open and collab.
And, uh, boom, you now have a collab environment in which to play with this.
Uh, by the way, if you can pretend you're a college student, you can get the pro plan of Gemini and Colette for free.
So, there you go. I'm gonna close that. Um… What's nice is, when I click on this, I'm going to open this in playground mode.
And so now, what happens is, I'm in playground mode, so changes will not be saved, which is great. I don't want to save this, I just want to demonstrate this.
And play with this. So now that I've got this in collab, let's go ahead and run some of these things. This is the command to install it. You literally do a curl.
pipe that into SH. And, uh, yeah, lots of places when you see commands like this, you'll also see a lot of comments saying, God, don't do that, it'll destroy your system, and, you know, the sky's gonna fall, and everything's gonna blow up, and, you know.
terrible things will happen. That's okay, this is Colette.
You know, if it did do that. I'm not worried about it.
Robert, could you just explain the, uh, 3-hour downage on the internet the other day?
So, I've installed…
Uh, probably. I don't know about everybody else, my internet was working fine. It's, you know, I basically got everybody else off so I can use the internet finally.
There we go!
So… Um… oh.
So, I've installed DuckD, it's installed locally. Again, going back here on the terminal, if I wanted to, I can run all these commands from the terminal.
Uh, let me just go back here and copy.
It installs it in an odd place. Um, oh yeah, let's just do this.
Da-da-da, put that in there. Uh, let's do paste… that export my path, and now I can go DuckDB.
And help, and boom, I've got DuckTV. I can… I can do all the fun stuff that I want to do with DuckTV.
Everything that I'm showing you in the notebook, you can also do at the terminal.
Right, so right within here, you just type DuckDB.
And then type in whatever you want to do.
I'm gonna stay in the terminal, or excuse me, in the notebook, just because everything is nice and how would you say, there's a nice history of what I've done.
So, let's go ahead and use DuckDB to query a CSV file, right? So… This is kind of the SQLite way of doing it. You download the CSV file, boom, there it is.
Uh, you then import that. into serial light. In this case, I'm going to use DuckDB to just query the database.
And I can say select count from flights, and uh, hey, I've got 3 flights.
Uh, let's select everything from Flight CSV. And there it is. Notice that all I have to do is I just specify the file name.
I don't have to bring it in first and, you know, say, hey, import this as this table. I just literally give the file name.
as my table name. Um, and uh, yeah, it's got this nice, uh, printing feature, if you will.
The cool thing is, is I don't have to download it.
DevTB actually has an HTTP, HTTPFS. Um, module that you install.
And instead of giving it the file name, you give it the URL to where the data is.
And it handles all the stuff behind the scenes. Here's the same thing, I'm counting it.
here I am basically going out on the internet, downloading it, uh, how would you say, behind-the-scenes, downloading it and querying it.
This is really useful when you want to get to some of the data that's in the U.S. Census, right? So here's a… here's a big, long URL to get some data from the census.
Um… um… Yeah. We're gonna query census.gov.
A couple things about this. Number one is, is just because things are CSV doesn't necessarily mean it's actually text. It could be some kind of other.
I would say normal ASCII text. It can be encoded as something else.
Ducky knows how to handle that encoding. If you know what it is.
Um, I think it does a pretty good job of guessing, but sometimes it doesn't guess right.
Um, so I had to go figure out what it is, but if I say encoding equals this, um, I can use this function.
this readcsv function, and this will automatically take this converted, or how would you say, decode it appropriately.
And provide that as a table to my SQL query. And if I do that, basically I can download the data, chug, chug chug.
While we're waiting, could you repeat that last sentence? You said.
This will just automatically decode it. to a table…
Okay. It treats it as a table, right? So you don't have to, again, like, in SQLite, you have to download it, and then.
import it, and make it a SQLite database. With DuckDB, you don't have to make it a DuckDB database. It just queries.
Okay.
the data directly. That isn't quite true. But it's close enough, right? In other words, you're not creating a DuckDB database, um.
in, uh, using this. If you want to, you can.
Right? So you can… you can use DuckDB as an ETL kind of process, where you get data in one format.
And then save it as another format. So, for example, read it as a CSV file, and then output it as.
An Excel file, or a Parquet file, or something else.
DuckDB can do that. But in this case, I'm not.
So… Here we go, uh, just queried how many lines we've got, 1157.
Um… one of the interesting things about it is that if we query it.
And my apologies, there we go. Um, it seems like that second time went a lot faster than the first time. Notice what… notice the data, though.
Um, and this is one of the things I love to show my students, is just because the data's out there in a certain format doesn't mean it's quality data.
Um, we always like to believe that there's a single thing for the header. It turns out that this, uh, this particular format has two lines for the header.
And when we query the data, um, it will just give us the data that we want.
watch, right? So then we have an offset of 1. Uh, and if we wanted to, we can query all this data.
So that's CSV. It also supports Excel files, and when I say that, I mean a modern ones, the XLSX files are supported, the XLS.
files themselves, the old version, XLS, they're not supported.
Again, we'll do it the old-fashioned way. We're going to download this.
I download this, uh, from the U.S. Census, and then I'm going to query it. Notice we… there's a, uh, um, an Excel plugin.
And again, I can use this. the XLS file.
in the first 20, and uh… Yeah, again, we're not talking about very well-formatted data. It's, hey, it's in Excel, but it's not necessarily tabular data.
In the normal sense. Well, that's kind of is a bummer.
So, if you had normally formatted data, you can read it even when it's not formatted, you can at least bring it in deductee.
Um, uh, to, uh, look at it. And figure out exactly what the headers are, and any kind of cleanup that you may need to do, or restructuring that you may need to do to query it.
Right? We're gonna do the same thing and query directly.
Just like before, with CSV files. Um, I picked a different Excel file. We don't have to download it first, we can just go straight out to the internet and query Excel files and pull it in. Uh, by default.
It pulls in the first sheet, this happens to be the first sheet. If you want to, you can select the sheet that you want. So here it is, I'm selecting sheet 1.
There is same data. This Excel file happens to have 3 sheets.
If I want to, I can select sheet 3.
Excel file… well… I can use DuckDB to get the data out of the Excel file.
It works better when it's actually in tabular format.
Um, but nevertheless, you can use this to extract data from Excel files.
Another format that's very common is JSON. Here we are, we're gonna download this JSON.
There we go. Um… We're gonna look at what it looks like. Uh, it's basically, uh, you know, column value, column value, an array, a key value pairs, right? So this is not that different from a table, uh, where things are now, um.
turned into key-value pairs instead of into rows and columns.
Dr. B will read JSON just fine. There you go. It's now basically turned this JSON structure into this tabular structure.
Um, if we have a bigger data set. Um, I happen to have one here, uh, this is the, uh, incidents from Albuquerque Police Department.
I'm just going to select a few of them, parse them first with, uh.
with JQ, just to show what they look like. Um… These are the first few. We've got incidents, or I think they're called… yeah.
Uh, they're called features, actually, in the JSON. Well, it shows us.
basically some attributes, right, so… This is some identifier for that incident. This is where it happened in terms of an address, the block address.
Uh, the incident type, this happened to be a wanted person, and the date and time of when the incident occurred.
We've also got some geometry information in here. So, uh, latitude and, uh, and longitude.
Um… We can actually unnest that, or pull it out into their own individual field. So, uh, straight from the JSON file.
So if I do that, we now have everything as a table… as a table, right? I've got the… the, uh, the ID number, the block address, and so on and so forth. Even things that didn't actually have a geometry, um.
Uh, uh, uh, uh… geolocation, latitude and longitude, um, it… it just leaves it as null.
Now, the downside is, is, um. I couldn't figure out how to get it to work with deeper nesting. Everything I read said it could do it. I couldn't get the syntax right.
But, uh, there's… there is a way, apparently, to… if you've got deeply nested.
You have to go through an unnest it and turn that into tabular data that you can then work with.
I just couldn't figure out any time for, uh, for this, uh, for this presentation.
Questions? Alright.
One of my favorite formats. is parquet. I like it. And the reason why is because if we go back up here, if we look at our CSV file, uh, that we have, and even our Excel file.
Um… the data types, it just… it just makes a guess.
Right? Bar chart, double bar chart, so forth. Um… let me go back up to here to our CSV file. I think that's what this was, right?
Yeah, it's, uh… notice what it's done, right? This is VarChart, okay? This is VAR chart, okay, bar chart, so on and so forth, but pretty much everything is bar chart.
Um, even the things that are floating point. Or coming in as Vartro.
My guess is probably because there's some values in there that look like this. I'm going to assume that that is their symbol for null.
Wonderful that they do that, instead of just making it a null. Um… Not a big fan of CSV files, not a big fan of Excel files, simply because we don't have data types.
Parquet, on the other hand, we do. So, where's my part A?
There you go. We actually have, yes, by design, that is our chart, but this is an actual timestamp.
Uh, date time stamp, I should say. This is just synthetic data that I made up for an exercise for my class, but… if we have these kinds of things, if we have data in a parquet format, we get to keep and preserve the data types.
Um, and DuckDB knows how to interpret the data types from Parquet format and query it.
Um, that's really cool, because when we get to things like geospatial files, and yes, DuckDB knows how to deal with geospatial information.
So I'm gonna download… this is the track information.
zip file, which inside it contains a bunch of geospatial files, including a shapefile, and if I read it.
Um, you can read that file. And it gives me all the information. So we've got a bunch of bar charts, which is fine.
Um, is what it is. But then we have these… the geospatial stuff on the far end, and I'm trying to slowly move over.
To see it. Alright, so we've got integers, right, so specific data types.
Uh, and if we scroll over here, these are all the shapes of all the different census tracts.
Uh, in New Mexico. And I'm trying to get to the… It's probably gonna be a while.
Actually, let's do it the old-fashioned way. Let's do it that way.
There we go. Um, if I do limit zero, it just gives me the call names along with their data types. Notice we actually have a geometry data type.
Right? So this is very common in spatial, uh, spatial work, where you can do things like, hey, I've got a point, or I've got a polygon, or I've got a line segment, or, you know, all kinds of fun stuff. I can now use DuckDB to actually do some further stuff and say.
Let's imagine I've got one… One table with a bunch of points, and another one with a bunch of areas, I could say, okay, great, let's join those based on which points are inside what area.
Right? Um, and do those kinds of queries. Or, hey, I want all points.
that are within… a certain distance of a particular area.
Or within a particular distance of the area that is created by a line segment.
Things like that. Um…
Well, why do I have the feeling, if I asked you to demonstrate that, this would be a entirely new presentation?
This would be an entirely new presentation, but that's one of the things I'm working on, because that's exactly one of the things that came out of the talk yesterday, is a bunch of people are trying to do geospatial stuff.
geospatial stuff is huge for, uh, for government agencies, right? It's everything from.
reporting traffic accidents from reporting. Uh, you know, hey, where do, where do roads need to be repaired? Where are buildings, uh, that.
I need to be, uh, um, you know, inspected or, you know.
all kinds of fun stuff, right? And so… I'm… you know, I can't speak for other states.
Um, at least not yet, but I would not be surprised if, uh, if every state is similar to New Mexico to some extent, in the sense of.
There's a lot of data out there, and uh… A lot of people… How would you say, a lot of people… There are some very talented people in government, and unfortunately, they are leading.
Now, um, either… either forcibly or, uh, or incentivized.
And, uh, and that amazing knowledge is going away with them.
So, um… Yeah, and these are some really cool tools that would be nice to show our government officials to say, hey, we could do some really cool stuff.
with the data that you have. So, yeah, that'll be something in the future. Um, one thing I couldn't figure out with this is, uh, also, like with the other ones, I should be able to just take this HTTP.
this URL for the, uh, for the shapefile, and put it right in here to use it.
But, you know, the documentation, AI, everything said, oh, you should be able to do this and this and this, and I couldn't get it to work, so… Um, did the old quick and dirty, downloaded it.
loaded it in, and that works just fine. So… that's it on DuckDB for now. Um, yes, I'd love to give a future talk on some other really cool things you can do.
like I said, it's power is in analytics. basically querying things and asking all kinds of questions. Um… For those of you who are sysadmins, or DevOps, or whatever the common term is these days, um…
log files, um, if they're formatted in the right way.
Um, you can do all kinds of cool things.
Um, if you want to go through and look at, uh, um.
I've got some old code that used. shoot, what was it called?
There's basically a utility that gives you, um. every minute it tells you, you know, what the CPU usage is, and uh… memory usage, and disk I.O, and a bunch of other stuff.
And I can't remember it off the top of my head.
Graphic or text-based?
But you can capture it. Uh, tech space.
SAR?
SAR, that's it. Thank you. So, with SAR, you can output a bunch of, you know, you can output a bunch of data, and uh… but that's what it does. It does point-in-time collection of data.
And, uh, and I, you know, back in the day, I used to save that to files, um, or off to other places.
Um, but then we had to parse it and put it into databases to query it.
Um, but technically, you don't need a database. You can use DuckDB and just.
basically point at the file itself and say, hey, can I query… can you query this and tell me what's going on? You know, yeah.
How often do I see spikes? You know, hey, do I see any… do I see any IPs that are, uh, unusual?
you know, high activity coming, you know, coming from strange places and that kind of stuff. Um, DuckDB is great for that kind of analytics.
Um, it could… again, what I didn't show is.
or what I did show is I… I showed you can pull stuff from CSV files, Excel files, JSON files.
parking and so forth. If you can pull it from those different disparate sources, you could actually join them, even if they are disparate sources. So you could literally pull in one table from a CSV file and another one from an Excel file, and say, hey, join on this column.
And, uh, and show me what's, uh, what's common between the two.
Question.
So… yeah, question.
Um, I thought the files you were pointing at were…
available via the web, but I'm…
kind of multitasking, was I incorrect about that? And the reason I ask is…
If you want DuckDB to work on SAR files, do you have to publish them, or do you run a local instance of DuckDB, or…?
would.
Okay.
Uh, local instance of duct EB. Um… Uh, at the… is SAR the name of the package?
That is the package.
Let's try it, let's update and install. And it's been a while since I've run star, so if someone whoops… Clearly, that is not how you spell it.
CIS stat.
is the package its problem.
Yep.
Oh, is this that?
Now… but that sounds right. Maybe I have to do an update first, and… Dude.
Uh, but in essence, yes, you could run sysad, have it saved.
you know, to files. If I remember correctly, you… SYSTAT would automatically… how would you say you would put it in… I think in Kron or something like that, and uh… and you can then do log rotation, right? So you can say, hey, you know, every…
Every 10 minutes or so, or 5 minutes or so, automatically rotate the log, right? And it would start a new one. So you then have all these.
our, you know, how many say, uh, older versions of the, uh, of your log files.
Uh, from SysCat, you can then go through and.
Basically, look at them, parse them, and analyze them.
SAR…
or other pieces in the sysstat package.
cron jobs collecting or Contemporary method is a SystemD timer file.
Uh, by default, once a day,
it starts collecting to a new file, and I think it keeps…
a month worth, I don't know if it's, like,
28, 29, 30, 31…
days aware, or how it knows the month, but for the sake of discussion, say it keeps
30 days worth.
you can give the SAR command,
Mm-hmm.
pointers to the specific files, and you can copy those off, and they are a day's worth of data.
Mm-hmm. And if I remember correctly, it also… it puts the timestamp for every regular as well.
Yes. Yep.
Does that sound right? Right? Um… Yeah, so you have all these things, you have the time.
You have a bunch of this information, um. I remember the format was not, uh… I think it… oh, was it Tabbed Limited, by default?
I don't know, and most of the files, it's actually multiple thing…
multiple data sets in one file.
So, it is text, it is parsable,
Huh?
But you've got memory, you've got processes, you've got disk, you've got network, you've got a whole bunch of different things.
all combined in one file, so it's like the first
Yep.
thousand lines, I'm pulling a number out of you nowhere, are one thing, the next thousand, or the next thing,
the next thousand or the third thing.
And so, parsing it is slightly annoying, but it is imminently possible to do.
Um…
But if I remember correctly, um, you can set it up so that you actually have different processes collecting the different information, so each of that goes into different files.
In other words, yes, there's a way to get everything in one file, but there's also a way… the options you can say is.
you know, I only want to collect CPU information and save that to this file. Disk I.O, I want to collect that and put it into another file.
Okay, um, I wonder if you're thinking it's collecting automatically, and then
I want to output the specific information, and I'm sending it to my CPU file, and I want to send network information over. But it's an output from the aggregate.
But it has commands to get just those things.
Yeah.
Yes.
entirely possible. It's been a while. But the data is there, the data is available, and so if you wanted to just get I.O.
You can just get I.O. If you want just CPU, you can get just CPU and so forth.
Yep.
Robert, there's a comment in, uh, chat from Brian.
Uh, spell Cystat with S-Y-S-S-T-A-T.
Give it a shot.
I've been bitten by that one before.
Hey, there we go!
That's why… When I used to do this, I used to have all this stuff in.
files, and I just say, this is what I need, just… Install it.
I'm a huge fan of IEC, so…
Mm-hmm.
now, how do I… God, does anyone want to guide me and tell me how do I…
it's probably a contemporary system. You may need to start the sysstat service, and it's probably…
I mean, I'm… yeah, um…
do system…
Oh, good gosh. Oh, probably not.
System control status,
Sysp?
CTL.
Yeah, it's not gonna be. Not in a doctor container.
Oh, okay.
Right. Because I probably don't have true access to System D.
Well, you can run…
If you look at the timer file, um…
It basically, it runs one command, like, SI underscore 1.
at whatever interval, and that does a collection.
Um, depending on the distro, it either is a
5-minute, 10-minute, 20-minute, 30-minute
default, and this is actually across Linux, BSDs, AIX, Solaris.
It's multi-platform, which is real nice.
Um, and for troubleshooting some things on some systems, I've actually…
Yep.
done cron jobs that are 5 stars, and the collection.
And then, at the end of the day,
you do SA underscore 2,
And it packages the day's worth of data up
rotates its own log file.
this actually goes back and predates
common availability of log rotate.
Yep.
No, Gary, you do not hear a presentation.
So… All right. Yeah, lots of things that can be done. Um, if you… if you can get it into some kind of a format, you know, obviously.
JSON, CSV. Right? Parquet and so forth. Um, you can then use DefDB to, uh, to parse it.
Um, I'm also looking at… Let me go to the documentation real quick, which I have.
Um… The link to the documentation itself.
Uh, let's see, here we go. Yeah, so these are all the different formats that you can get data from. So yes, you can actually use SQLDB.
to query MySQL and Postgres databases. You can use DuckDB to query SQLite databases.
Um, as well as many… These other formats, um, I'm looking for S3.
I'm guessing that's what AWS is. And that's what… I think that's what they mean. So, lots of different ways to, uh, to, uh.
Uh, query data and do some analytics on it.
If you don't want to use it at the command line.
We've got a bunch of… they've got a bunch of APIs that you can use.
Python is one of them, R is one of them, so that makes it really nice, at least for our work. Uh, WebAssembly is another one, for those of you who like to work just in a browser.
And then, of course, they've got C, C++, Go, Rust, and so forth.
Lots of different ways to play with it.
All right, any other questions?
Alright, if not, let's call it a wrap. Thank you, everyone. Appreciate your time. Hope it was useful.
Oh, fire away.
I got a question. This goes back to the, uh… To the first topic.
Huh?
You were talking about the, uh, curious… Things that you could run. Um… Uh… Pearl? Really? Pearl?
I've greenfielded Pearl within the last month.
Sorry. When it comes to scripting languages, taste and elegance are my touchstones.
Yeah.
Um,
So I imagine there's other than Pearl, right? You've got… Besides Python.
Am I right?
Um, well, so, what's nice is, in those environments… let me actually go back to this guy here.
Let me go back to the terminal, um… Let me see, I'll just pick one out of the hat. I don't know. Ruby, perhaps. Um, Ruby isn't… Where were you?
Yay!
Oh, boy. Ruby isn't there, but what is nice is we can install it.
Right, so we could do ruby-, you know, install Ruby, um… And I'm trying to remember, what's the name of the, um, interactive, um… Is it IRD? No, IRB?
The IRB… Interactive Ruby.
Therapy. Is that the favorite one?
Well, no, probably is better, but… PRY?
Brian, let's try. Let's see if they've got that.
Yeah. Better that off. And let's see, do we have Pry? And there we go!
Uh, puts… quote, hello, world, quote, parentheses. undefined outputs.
Yeah, that's right, but… and you don't need the parentheses, yeah.
There we go! You don't need the parentheses?
Oh my god! It's turning into old version of Python.
Not in Ruby. You can put them in if you want, but…
But my point, though, is that, uh, is that a… is that just something you're running in the terminal, or… Is that, uh, like a… A first class… Um… thing that the… Uh… Jupyter Notebook can do.
Oh, good question. So… Jupyter Notebook, yes. Colab, no. Unfortunately, Colab is, uh… What's the word I'm looking for? Controlled.
I guess is a close enough term. It's controlled by Google, and if we click on the runtime.
types, uh, we basically only have 3 options. We can run Python, we can run R, we can run Julia.
Now, that said. Jupiter itself has a bunch of different kernels that it can run.
Um, and so, this is a… Great example of what you could do is you could fire up a code space.
Run your, you know, install your own Jupyter environment, include the kernel for Ruby, and now you've got a Jupyter notebook that will run Ruby.
Oh. Okay.
So… So, that's a great example of a limitation of Colab, and that's also, of course, that's true of Kaggle. Kaggle only runs, I think, R and Python.
And only a particular version. But if you want to have more control, CoachBase is a much better environment.
to do that.
And when you're running the, um… that you put a notebook in Codespaces?
Right, um… you're still getting all the output.
but is going through your browser, coming from your browser to your browser.
Right? In other words, the output of Jupyter Notebooks is.
showing up in your browser. Ah, okay. That's what I was asking.
Correct.
Okay.
Um,
Yep. Yeah, it's a fully browser environment. Um, if you want to… If you run it at the command line.
I don't know… I mean, there's… IPython. Right? Which was the original.
Mm-hmm.
command line version, which is truly just terminal. terminal kinds of stuff. Um, but I think Jupiter is strictly.
browser-based.
what if you did some of the not-quite-supported
Um, log… get it up and running in the terminal, and then SSH out with, uh…
tunnel or something…
you know, questionable color hat.
reverse shell-type thing.
that you could then access that from your traditional terminal of preference.
and be inside the running…
notebook, but…
outside of the browser.
Does that make sense?
I… I'm trying to see… I am… I understand the part about the reverse tunneling, um, but I don't see how you would be able to run something.
So…
You start the notebook…
like your… where you are right now,
like a Jupyter Note outside of the brown… I mean, you… Yeah, I don't…
and make an outbound connection to something
that could then allow reverse connections in,
or remote control, or something in. So, say, if you were visually impaired, or what just really preferred to use a different non-web browser interface,
You come in through that
connection you create with the outbound connection,
from whatever terminal, emulator, maybe you have a VT100 hooked up to something,
And you just want to play with the Jupyter Notebook,
off the glass teletype.
Is that where you are trying to get, type of thing, Ed?
outside of the web browser?
Ed, you there?
Um,
Oh, I'm sorry, I was looking at DuckDB, sorry, but…
I am kind of assuming you're wanting to run some things from your terminal
where it is an environment more conducive to your usage patterns.
Correct.
No, actually, uh… My usage patterns.
Are… aren't like programming and stuff, yeah, terminal type stuff is better, but… For, um, any sort of… Thing like this, then the bro… Blizzard is better. The only problem is.
This is that, uh, when I… When Robert and I were.
Looking at this some time ago. Um… I went to his thing that he set up, I think it was a collab thing.
And, uh, I found out that… The… some websites are more or less accessible than others.
Right? And Colab was one of those ones where.
There's a… there's a few gotchas when it came to accessibility.
Now, that was, like, how far… that was, like, 2 years ago or something, wasn't it, Robert?
That's a while ago, so they might have been… Yeah, they probably might have approved it by now.
It was a while ago.
One of the two.
Yeah, right. So… Yeah, so, to answer your question, uh… I think I'd prefer the browser if it's accessible.
Okay.
But, uh, the DuckTV thing… Uh, I could have swore that it was written in some, like, esoteric programming language, but according to my.
Research it written in C++.
Probably started written… it was… it was probably originally written in scheme.
Oh, could be, yeah, I don't know. But, uh, I thought it was, uh, they, um… No, I'm thinking of, like, uh… TigerDB or Beetle, or something.
That was the other one, I guess it was written in Zig or something.
Right. That's supposed to be very high performance.
Like, you could do, like, a billion records in under 10 seconds, that type.
Hey, I don't know.
Yeah, DuckDB can do as… is very performant as well.
Again, it uses some tricks. To do that kind of stuff, right? So…
Mm-hmm.
But it's not a… it's not a transaction database, right? It's… it's… and so, therefore, it's a columnar database, and therefore, it's a lot… You know, it can find things very quickly, um… And, uh, and do the analytics on it very quickly.
But you're not… you can't do, like, insert or update.
Statements…
It's not that you can't. It's just that, uh, that's not really what it's meant to do.
Right? And I almost want to say, that may also be giving.
that… my knowledge on that may also be getting, um… I would say dated, because I was looking at a few other things, and they're now saying, oh, hey, we should make, uh, DuckDB.
asset compliant and all this other kind of stuff, and it's like… Well, that kind of… kind of… defeats the purpose of making it, uh, you know.
an analytics tool. But… The short answer is yes, you can do inserts and deletes and things like that, but that's not really what it's designed for.
Okay.
Any other questions?
All right. Then, uh…
I got a quick question, so you… probably been basing some of this on, uh, what you've been doing for.
for your classes, uh… What, uh… What's the… class title, or the type of students.
That, uh, that you're exposing to, uh. To these things.
Oh, boy, what kind of… I get a very broad range of students, um, all different kinds of range… all different kinds of ages, all different kinds of backgrounds.
Um, um… you know, I get people who are, uh… Um, how would you say, fresh out of high school?
Uh, and not… not in college, all the way to people who have, uh, you know, advanced degrees.
Mm-hmm.
Um, on some of this stuff. Um, in fact, it was kind of fun. I had a, you know, we… One of the topics we cover is, uh, uh, neural networks.
Right? So, um… Multilayer perceptrons, convolutional neural networks, etc, that kind of stuff. And, uh, you know.
in my introductory slides about the topic is, you know, these neural networks are basically patterned after the neurons in the brain.
Um, and, um, I had a retired, um. neurobiology professor in my class, and she kind of raised her hand and.
And, uh, it said, that's not really how the brain works.
And I said, I know, I know, but it's not nearly as exciting if you say, we're modeling this on our kneecaps, you know, on reflex responses, and so.
Yeah, it's much more… it's much sexier to say we're modeling it on our brain.
Um, so… Yeah, so, yeah, very, very broad range of students, um.
I would say that the most. the students that get the most out of the class are students who have been out in the field for a while.
And when I say the field, I mean anything.
Um, you know, I've had, uh, um, I've had.
uh, uh… The police officers take the class. Currently, I've got a firefighter taking the class. I've had landscape architects, I've had, uh, I've had software engineers.
Actually, I've got a software engineer right now in my class. I've got people who are data architects, and all kinds… All different kinds of people who are taking this class, and uh… it's kind of neat to see.
Uh, um, you know. then understand now how to work with data, right? Um, data's becoming such.
so prevalent in so many different fields. Then it's like, oh, wow, I could do this with, you know, the data that we've got and, you know, do all kinds of fun stuff, right? So, um… Yeah.
How…
accurate would it be?
to walk away with…
the people that are getting the most out of your class,
are people that have been in the field, exposed to things, seen raw
data have wondered if they could ask questions, and you're showing them some tools…
to turn that data into useful information
and you're seeing the light bulb come on to them and starting… you can see they think,
We could get this, we could think about that, we could learn this from it, and…
They've had the exposure and the need.
Those are the students who do incredibly well. Right? Um, and it… Yeah, it's… the ones who… The ones who've been out in the field and have been working with.
data, and they've been doing it in Excel, for example, or, you know.
even by hand, in many cases. Or they're using specialized tools.
That, uh, that allow them to do, you know, a certain piece, but not a whole bunch of other pieces that they'd like to do.
Um, they're the ones that come and, you know, their mindset is already, there's got to be a better way. What's the better way? There's got to be an easier way. There's got to be tools that help me do this kind of stuff, right?
Um, they're the ones, you know, they've… they're looking for the tool, they're looking for the way to do things, and how to… and of course.
how to ask the questions about the data that they're getting.
Um, those… Those are the ones that, uh… yeah, it is… it's really a joy to see the light bulb go off.
Um, uh, in their mind. I had a flow cytometrist.
Uh, who has taken this class, and, um, flow cytometry is, uh.
Uh, in the clinical lab, these are people who get blood samples.
Um, and uh, and they run them through this, uh, this really small hole, or excuse me, very, very small channel.
where only one cell can go through at a time, and uh… they mix the cells with a bunch of fluorophores, right? So, when light hits it, when a laser hits, it actually, uh, it fluorescents in different wavelengths, depending on the different.
Markers that are on top of the cells. They collect that data, and then they plot it, and they look for different clustering stuff. One of the things we teach is, you know, different clustering algorithms.
And, uh… Uh, he looked at one of the, you know, I was talking about one of the cluster methods that, uh.
And he goes, well, wait a second, can I show my screen?
And he shared the screen, he goes, this is what I do all day long.
You know, and it was showing us these different clusters and patterns of these flow cytometry output. And he goes, and I've got to do this and this and this and this, and this with it. Are you saying there's… there are actually tools that can help you.
go through this and organize this much more quickly.
You know, and it's like, yeah, if you got the raw data, you know, be happy to sit down, we go through it.
So, yeah, that was… that was a really cool light bulb moment.
Um… It's neat to see. It really is.
Do you bring any, uh, LLMs into the data analysis part?
do I bring them in? Um… So, LLMs are kind of interesting.
using them, especially if they're pre-trained, right? So, you know, I encourage my students to use.
Gemini and Chat, GPT, and uh, you know, Notebook LLM, and so on, right? I encourage them to use it. I also show them how to, uh, you know, some of the fun stuff.
Uh, we, you know, I try to encourage them to actually create hallucinations. I show them a few hallucinations that.
that I come across. Um, so there's… there's that using LLMs kind of facet.
Um, the other extreme is actually building an LLM, you know, actually building a neural network and building a transformer now.
Excuse me, all the other pieces that go into a full, um.
LLM thing. The hard part about doing that is.
Realistically, we just… we don't have the resources to do that.
Right? We can… we can do some very small things. I can show them how to do, you know, neural networks.
Hmm…
But even, you know, even a relatively small convolutional neural network, I mean, we blow through.
Uh, the, uh, the GPU cycles that CoLab provides us.
Right? Um, usually a few times, uh, in the last… we leave, uh, neural networks for the last module.
It is not uncommon to blow through them a few times before finally getting a neural network that works.
And the neural network that we built as a project is nothing compared to.
a transformer. I mean, it's… it's miniscule. Right? So…
Right. Yeah, but I was just wondering if you used, like, um… Uh, sysadmin data, like you were talking about earlier.
You know, and if you were looking at the… log out from your HTTP server.
Yep.
And you say, hey, you know, uh… Tell me any, like, hack attempts from North Korea or something.
Right? Like, you know, like, use an LLM to summarize that.
And, uh, but you don't go that far, probably.
That's what you're saying.
We… we… we don't, um, and…
Yeah, how to say this? that's kind of cool to do with an LLM, but at the same time.
That specific kind of problem is. probably something that we can do with, you know.
what I'll call plain old machine learning models. Um… And so, you know, LLMs are computationally expensive, um, and they're not… they're also not deterministic, right? So, um… They're good, they're good for a lot of things.
Right.
Um, it… where I think an LLM would be good for is to give it a sample data set and say, hey, what tools would be appropriate to.
analyze this kind of information. You know, would a, you know, some kind of a clustering method be good, or some kind of a, you know.
Um, a classification or regression or some other kind of, you know, traditional machine learning model would be, you know, might be more appropriate.
And happy LLM tell me about that. I think that would be, uh, probably… a better use of an LLM.
Um, but… who knows? You know, maybe, maybe there are LMs out there that are trained specifically for that kind of stuff.
And, uh, um… You know, you can throw an LLM at, you know, and maybe it turns out that you can make it a very simple LLM, relatively simple LLM.
You know, very, very small, and have it compute.
Vincent had a question in the chat.
Something to try.
Yeah. Yeah, Vince, I believe it's Vincent that asked, uh, has, uh, have, uh, have I, have I had plenty of journalists in my classes? And the answer is, uh.
Uh, no, unfortunately not. Um, I think it'd be great, especially for investigative journalists, uh, looking, you know, looking at tons and tons of different, uh, data sources.
We tried to do some things with the, uh, with the local newspaper here.
And, uh, um… Timing just didn't work out, um… Uh, I would say we… they were undergoing some changes, and we were undergoing some changes here, which kind of sucked away our time from actually making something useful.
But, uh, some of the projects that they proposed that, uh.
Um, we're right up our alley. It would have been great.
Um, so yeah. That would be cool if I could get journalists. I think they would benefit tremendously from this class.
Um, especially with querying data, pulling data from different sources, and looking for patterns and trends and things like that.
Um, Eric has a similar question. Have you had anyone in law in your classes.
Uh, the…
I think… I don't think so. I… you know, I've had a CPA.
Uh, she… she was… she was like, oh my god, all this stuff we can do with this data, um… You know, again.
She dealt… she was dealing with lots and lots and lots of data, but in a different context, and again, it's the, hey, there's got to be a way to, you know, there's got to be manual, or how would you say, better, more automated way to find patterns and things like that in the data that we're dealing with.
And, uh, when, again, when she was going through this class, like, oh my god, I could use this, oh my god, I could use that. Oh, I could do this with that. So, yeah, lots of, lots of cool things from that perspective.
Um… A lawyer or anyone in the law field.
not that I know of has been in this class. What I have had is I've had at least 2 students of mine.
actually end up working, um. either in the, uh, either in a law firm or in, um… Uh, shoot, what is it called? The… the lawyers that government people have. Um, I don't want to say it was… it might be the district attorney's office or something like that.
And so, uh, yeah. Uh, and so, to answer your question, have you dealt with PHI data issues?
Uh, we're actually working with… Yes and no, um… We are currently working… one of the, uh, one of the teams in my current cohort is, uh, is working with.
Um, PII. Right? Um, and uh, it's not PHI, but it's definitely protected, uh, um, our personal… personally identifiable information, and so that data needs to be scrubbed.
And, uh, and modified in all kinds of fun ways. Um, and so that's kind of an interesting project.
That one is actually working well. Uh, there was another, another cohort that tried to do something similar in the past, uh, and unfortunately, the.
The… when, uh… On the, uh, how would you say, when the organization that was providing the data discovered that, hey, we're going to do a presentation on this in front of a bunch of people.
Uh, they kind of pulled the rug out from it, said, no, no, no, no, that's our, that's our data, and it's too sensitive, and we can't let it go out.
Right? Um… Uh, have I dealt with open source tools with on-site data that is, uh.
Uh, vendor format? Um… Sort of. Uh, so one of the cool things.
Um, with, uh, with pandas. Uh, Pandas is a, uh, um.
Uh, basically a tabular data format, um, that you use within Python.
And, uh, pandas can actually read… there are a lot of.
What am I looking for? Um, uh, tools out there.
or libraries that allow you to read it from a particular format and turn it into a format that makes it tabular and workable.
Uh, for pandas. So, for example. Uh, pandas can read data from SAS, SAS-formatted files, SPSS files.
Um, uh, there is also… Um… Uh, what's the word, uh, um… There's some proprietary formats, and those proprietary formats have tools to convert it into some other format, like, let's say JSON. Then we have it in JSON, you can then use other tools, including pandas, to work with the data.
Um… So… yeah.
That kind of stuff. Um, EG is maybe no public API, etc.
I'm not sure what your… what the question is. Um… Yeah, there's a lot of data out there.
Uh, where there is not a… there is not a public API. So, for example, yesterday.
And this is… this is odd, and this is… you know, I'm going to talk to some folks during the intercession.
Um, New Mexico tracks traffic data, so traffic accidents, um.
You know, a bunch of information that goes with it. The obvious ones, date, time, location.
Um, uh, you know. Who's, you know, who's involved, how many vehicles involved, all that kind of fun stuff. Uh, that information is not publicly available.
Sort of. It is publicly available. Um, but you have to ask for it. There's no API, there's no, uh, there's no way to download it directly. You literally have to fill out a form.
and say, what data you want. And you have to submit it.
Then people review it, and then they give you the data where they don't.
And I'm not quite clear on why that's the case, but I'm hoping to resolve.
or I would say, get some clarity on that, um, during the intersession.
Um, so yeah, there's… there's… amazingly, there's a lot of public data, what I would consider public data, that is not publicly accessible unless you.
Number one, know to ask, and then ask for it.
Right? Um… puzzling, but… but… so it is.
Uh, going back to the version of the question about, uh, did you get journalists in the class?
our calendar, uh, we had put on, I believe it was the first week of August.
Uh, there was a class at University of Missouri at Columbia called Advanced Dating.
Yeah.
Advanced Data Journalism Boot Camp. Using the R language.
Uh… 5-day, I believe, class.
Yep.
So, uh… so, yeah. lawyers and journalists need to start learning this stuff.
Yeah, hello. Yeah, I had a look at that class, and I looked interesting. It's, uh, yeah, it's basically where the data sources are, how to get the data, how to query the data, how to work with it.
They're using R, which is… which is totally fine. Um… You know, I… the class that… I teach here used to be a combination of R and Python, and we decided to settle on Python for a few reasons. One is, um.
Just that Jose, Python is used in many other fields in addition to data science, and so if.
Uh, this… our… the goal of this program. is not for you to go out and.
go to college, right? This isn't, you know, hey, you can now take, uh, you know, another class at this school and further your education.
I mean, that's great if you do, but that's not the purpose of this class. The purpose of this class is to take people.
who either are transitioning from one career to another career, or, um.
They are, how would you say, wanting to upskill, right? So, again, the CPA, the… the, uh, the, um, um… the firefighters, the, the, the, uh, um, the software engineer and so forth, right? They want to learn new schools, uh, upscales.
To basically either get a promotion in their current job or go out and find another job, right? So that is our primary focus, is to get people employed.
Um, kind of second to that is for them to actually start their own companies, which a few of them have.
Um, which is… which has been kind of interesting to see, um, which is… you know, I think, wow, that's really cool that they did that. Um, so that's kind of the main thrust.
Um, and so therefore, we picked Python because, uh, you know.
a lot of times. They're not letting… you know, how would you say the first jog, especially if they've come from something.
you know, a completely different field. Uh, going out and doing a data job is… is… Uh, what would you say, usually not their first step. They're going to be doing something else. And so, uh, Python…
is something that's a little bit more in demand for more entry-level kinds of things than, uh, than R would be.
The second reason is, you know, I'm a firm believer that it's not really the language that you learn, but rather how you think about the language, the data structures, the algorithms.
that kind of more abstract kind of thinking. Uh, coming up with the questions, uh, all that is.
Not depending on the language that you learn. Right? Um, and so, therefore, um.
The thought process is what you really should be learning, and then if.
You find a job where they require art, you should be able to pick up a book on R and ask the questions, okay, I know how to do this in Python, how do I do the same thing in R?
Now, with the advent of, you know, these LLMs.
Um… That is becoming easier and easier, right? You know, you can literally paste, uh, hey, I did this in Python, how do I do the same thing in R? Um, and it'll write you the code.
And with these environments that I just showed you, you could literally try it out.
You could literally have two different environments side-by-side and say, hey, here it is at Python.
ChatGPT, tell me how to do this, oh, this is it, cut and paste, put it in, uh, put it in the other notebook at R and see if you get the same result. And if you don't, well.
There's no learning opportunity to figure out why that isn't the case, what actually is different.
Right? Um, but… The real process, the real thinking should happen if, how do I tackle problems? How do I, uh… how should I think… how should I be thinking about organizing the data so that I can ask questions?
Um, and that… You know, I mean, we could even be doing this in Ruby.
I like poking the bear.
Yeah. So, um, yeah, or… I mean, and I'm not kidding, I mean, we could be doing this. I don't know enough about Ruby, but… I'd like to believe that, uh, you know, these data structures, such as, you know, data frames.
I mean, Python didn't have data frames in the beginning, right? R did.
Uh, and so that's… that data structure actually was.
for lack of a better word, ported to Python.
Right? Um, so… I don't see any reason why that can't be ported into… into, uh, um, into Ruby, or Java, or C, or… You know, JavaScript, heaven forbid, but, you know.
It's possible, right? That's… that's what we should be thinking about, is what are the data structures, and what can we do.
with those data structures.
Yeah, it's, um… It, uh, is, uh… A native format for a new show.
Data for himself I'm talking about.
Uh, data frames are a native format in NuShow.
Oh, sorry, say that again?
Yes. So I understand, yep. I haven't played a new show in a while, just have not had the chance.
You know, so…
So… But it's kind of neat because of the… of the piping capability, right? You can… You know, you can take some, you know, you can slice and dice and pipe it from.
One thing to another, and… You know, and then write it back out as a data frame.
Or use it as a day, right?
Yep.
Which is… which is kind of one of, you know, kind of the.
neat things of doing a lot of this stuff in Python is that, you know, being an object-oriented language, if you will.
You can do things where you take a data frame and you, you know, instead of piping, you're doing, um.
Messaging, yeah.
Um, method chain, right? Right? So, you know, hey, take this dot, you know, take this column and add 5 to it, and then take that column and do it with this column, and bop, bop, you know, just…
Keep on… keep on going down. Lots of cool stuff there.
Um, in fact, you know, one of the cool things that the original.
creator of Pandas was a gentleman by the name of, um.
Um… shoot, I forgot his name now, Wes… I'll have to look it up real quick. And those…
Wes McKinney. Um… He's now making… he's now gone on to do other things, right? He's basically created a group, it's now, you know, Pandas is now maintained.
via, um, uh, what do you say, a consortium of a bunch of people. Uh, he's now working on other things that are related to it.
Uh, and if I'm not mistaken, he's working on Pi Arrow.
And, um, uh, or excuse me, I should say Apache Arrow, of which there is a, uh, um.
uh, you know, Python bindings for it, but Arrow is not… is not… limited to Python, right? So, anything should be able to, uh, to work with Arrow.
And so, in theory, yeah, Rust… Pearl, Ruby.
Whatever, right? Um… So… Lots of cool things going on out there.
It's happening, changing very fast. Um, Vincent asked another question, what do you know about the pioneers in analyzing large amounts of data, such as, uh, Wasley Leontif?
Actually, I don't, um. But there's probably a lot of things I don't know.
That's one of the cool things about doing this, is I learned something new every day.
Well, lots of… lots of cool stuff out there.
All right, any other questions, folks?
I'm just gonna make an observation that, uh. When I talked to Robert about doing the talk here this evening.
He was like, oh, well, you know, I've probably got maybe, you know, 50, 5-0 minutes of material here that, you know.
A few questions, yeah. It might run an hour, maybe, you know?
So, here we are… Uh, some 2 hours after Robert started speaking.
My apologies for rambling on.
And uh… so, to those of you who… Well, no, but the thing is, you know, I mean, so often when we talk to somebody about speaking, they're like, oh, well, I… I'd only have a couple minutes of material. I couldn't possibly, you know, speak for.
I don't know, more than 10, 15 minutes of time, and so there's no point in me talking, and.
This is a fine example that, yeah, uh, what you think is only a few minutes of material.
easily, uh, with, you know, when you add a little explanation to it and a few questions.
It blossoms, and people really enjoy it, and learn a lot from it.
Part of that is due to Robert's talent, but.
Any of you out there can probably wind up doing a lot more of a presentation than you realize, so… As usual, an offer, an opportunity.
A request from all of you to, uh, step up and do a presentation sometime.
Hey, Gary.
look in the mirror.
Yes. Oh, hello.
Yeah, what if… I mean, one of the other things that I've also pitched to Gary when we've had our phone call and conversations is, uh, is doing something called, uh.
Um, here, what did Scott Brandeman call him? They're not lightning togs, they're basically a series of vignettes tied together.
Is that Thunder Talks?
Uh-huh.
Um… Thunder talks, yes, or… Or Rolling Thunder or something like that. Um… You know, where you get, you know, maybe… two, three people together to talk about, uh, you know.
different, but oftentimes related topics, and uh, you know, again.
it only has to be a 5-minute presentation, that would be a lightning talk, right?
But, uh, if you had a Q&A to it, you know.
You're looking at maybe 10 minutes, 15 minutes, um… And you get 2 or 3 of those, you know.
You have a… you have some good… Discussion along with, uh, with some cool topics.
We've also done things that we used to call keyboard karaoke.
Mm-hmm.
Where we would have people just… I mean, we were face-to-face meetings.
We have people come up to the… To the projector terminal, and just start doing something interesting at the command line.
Or… whatever. And it's the same kind of thing.
Sort of.
I wonder how… how could we do that? Well, obviously… What comes to my mind of how to do that would be, you know, set up a, uh, you know, a shared.
X, uh, excuse me, a shared screen session, right?
That's really easy to do. You just fire up a machine, set up a screen, two people SSH into it, SSH des X, boom, you're both sharing.
Yeah.
You're sharing the same terminal at the same time.
Right? Um… I mean, we did that… God, what, what year are we in now? 2025? I mean, shoot, we were doing that… this back in 2020, or… 2000, 2001, I mean, shortly after I moved to St. Louis, you know. It's been around for a while.
Right.
Right? Um, it'd be cool to do something like that.
I don't know what newer technology exists for doing stuff like that. I mean, I've seen screen sharing, and I've seen it where people can.
take over somebody else's computer, right? You can… you can grant control to other people.
But, uh, nothing like… Nothing point like, you know, a shared screen X.
I don't… isn't there also… also a whiteboard in these systems?
But then again, I haven't…
Yeah, like… Yeah.
That people can scribble on? I mean, it's not the same, but that's kind of… Sharing.
Yeah.
Uh, college, the new fancy terminal multiplexer attitude. start up a web server.
So everyone can hop on, you know, their browser or whatever, and there's your terminal.
Yeah, let me see if I can't pull up the docs.
Not familiar with it.
It sounds like a good lighting presentation.
Yeah. One step forward!
Oh, wait, that shows up.
But seriously, you know, put it out on the mailing list, and uh, you know, with some information, and, uh, you know.
I'll be sure to have a look at it, and uh, you know, see how far we can go with it.
But that'd be kind of cool. Um, actually. Hey, Ed, are you still there?
Yeah, Ed's still on. Are you still… It might be stepped away.
Yeah.
He's still signed on.
Yeah, I see him, and I see his mic is open, but I don't think he's there. I, you know, I'm just… one of the cool things that… About a screen.
session, is that it's… it's all text, it's all terminal, right? So, um… You know, even Ed could, uh, could participate.
Okay. So… I mean, I… one of the cool things I remember seeing.
I make so many mistakes that, uh… That, uh, that, uh, people would be upset by the mistakes.
No, no, no, I think people should show their mistakes, that.
Absolutely. Show, you're human.
Agreed.
Yeah, right. Yeah, it showed mistakes, and then you have to show how you corrected the mistake.
And more people learn. from the corrections, and they do from just a bland presentation where no mistakes are made.
Um, if I can interject real quick, I have always said
Yeah.
You learn more from mistakes.
at least 4 times as much.
than just doing it right.
You learn what doesn't work,
You learn the symptoms of the wrong thing being done,
You learn how to undo it,
And you still get to learn the right way.
Maybe with some more iterations of the first three in there, so you learn more.
with a mistake than just doing it right.
And it calms the horses down.
I was just gonna say, though, that, uh… My mistakes are much more pronounced due to the fact that.
I can't see the screen, so I'm trying to kind of go by memory as to what I've typed before.
So, I don't think it would be that enjoyable for most people.
Fair enough.
I think it'd be something cool to experiment with, right? Because I know, Ed, you've got a thing that you made that… or have that talks back to you, right?
The thing that I have that talks back to me?
I mean, I just use the screen reader, is it?
His spirit. Wait, say that again?
I'm not sure I… I use the screen reader on the computer.
screen reader.
That's not what you're talking about, though.
Well, I'm not sure if… so if you're at a terminal and you're typing away, does it… it tells you what you're typing, right? So you… you've got that…
Right, yeah, I can either do, uh, character by character, or word by word, or line by line.
Okay.
Right. So, yeah, I mean, but you just… yeah, it does… it does repeat back to me.
That you're right about that. Uh, the problem is, is that… Um… I… I make a lot of mistakes.
So, I end up having to, uh, either edit the line I'm on, or just retype it.
Fair enough.
I'm better at it than I used to be, but… Still not that great. Yeah, I… you know, we could do something like that. I mean, if I had something to contribute in that regard.
Alright, I'm gonna take off too, guys. See ya.
Thanks, Brad. We'll see you next time.
All right, Ed, take care.
Stay warm!
Yeah.
Alright. Alright, folks, yeah, it's almost… I don't know what time it is over there, but it's getting late over here, so I gotta head on home.
Alright.
What are this week's winning lottery numbers?
Anybody have any questions before I ski-daddle?
Uh, let me, uh, stand, stand. Don't close it out yet, I'm gonna copy the links.
Well, thank you all very much for coming tonight. Thank you very much, Robert, for the excellent presentations. Kept us all.
Captivated, I'm sure, and uh… Uh, we appreciate everybody's, uh.
Uh, good comments and questions in there as well, so… Uh, thank you all for coming. I believe our next meeting, uh, is it the new users log?
I think? Stan, am I correct on that?
new Linux group is next Tuesday. I'd like to point out that we do archive the.
Let's…
from Zoom, we archive the chat. recordings. So they're available in our archives. The entire chat.
Okay. I think I got what I needed anyway.
is up there.
Got the… The question was about the cost to the homeowner. Okay, thank you.
Uh-huh.
Thank you, Eric. Alright.
Alright.
Any other comments, questions?
We can shut down the recording, but still continue with the meeting if anyone.
I think I suggest that we… this time, we stop the recording of the meeting for the archives.
Oh, really? Sounds good. This is a bookend, uh, the, uh… This was the, uh, Thursday, November the 20th, 2025.
The St. Louis Linux user group, the St. Louis LUG.
Our speaker tonight was Robert Sitek. And, uh, we did, uh… Basically, two presentations. Uh, one was on, uh.
Uh, the availability, the ubiquitous availability of Linuxes. You can sign into.
Over the cloud, and the other one was. On the use of, um… I'm blocking. What do we do?
DuckDB.
Talk to me, Dr. B. Thank you. Uh, so… Oh, excellent. Thank you all for coming tonight.