The Python Podcast.__init__

Share on social media:

Full show notes can be found at http://podcastinit.com/episode-1-thomas-hatch.html?utm_source=rss&utm_medium=rss

Brief Intro

• Hosts
• Overview
• Python at Chefconf!
• Plug for Talk Python To Me

Thomas Hatch Interview

Picks

• Thomas Hatch
  • Flow Based Programming
    • IOFlo

  
  
  • Imagine Dragons
  
  
  
  


• Chris Patti
  • Stone Imperial Russian Stout
  • Python One Liner Games
  • Boston Python User Group



• Tobias Macey
  • Noisli
  • CopyQ
  • Pelican
  • Moving From Heroku to AWS With Salt Part 1
  • Moving From Heroku to AWS With Salt Part 2

Closing Remarks

[00:00:15] Unknown:

Hello and welcome to podcast.   Init. We are recording today on April 8,   and we have Chris Patti and myself, Tobias Massey, as your hosts.   Today, we're interviewing Thomas Hatch, the creator of SaltStack.  

[00:00:30] Unknown:

So I just wanted to say a quick word about, I just came back from ChefConf in, Santa Clara, California last week. And I know this is a Python podcast, and, moreover, we're talking to the man behind SaltStack. So Chet might be just a little bit out of out of topic. But, I just had to say, on on the Python   front, it's amazing to see how much   the polyglot programming,   way of life has taken. There was a lot of Python talk in the in the hallway track there, And, some of it may spawn some interviews that we may have in upcoming episodes, and I'm really looking forward to. It's, it's neat to see people becoming more open minded about tools, and when you get smart people using interesting things, it's, only coolness can result.   So that was great. I had a great time. I highly recommend it.  

Also, on the community front, but back more towards Python, I just wanna mention that,   there's another podcast out there. It came out right around the same time as, you know, we kinda looked around and said, hey, there are no Python podcasts out there. We should make 1. And it's called talk Python to me. It's at   www.talkpython   to me.com,   and we'll have the full URL in the show notes. It's a great show, and, I just listened to episode 1 last night and this morning. He he talked about, and had an interview with Man Behind Eve,   a really cool,   REST API web service module for Python that just lets you take mongodb   or other database   schema and make it into a a restful API very easily. I look forward to playing with it myself, and, we're just happy to have someone else in the in the game here. I think more is better in this undertaking.  

And so,   please go give him a listen. You you won't regret it.  

[00:02:19] Unknown:

Alright. And with that, I would like to introduce our guest, Thomas Hatch. Thank you for joining us. And would you please, introduce yourself and explain a little bit about your background to our audience to anyone who may not have heard of UC yet?  

[00:02:33] Unknown:

Sure. And thanks for having me.   Yeah. My name is Thomas Hatch, and,   I'm the original author and founder of the SaltStack Company and, author of Salt.   Yeah. I started working on Salt back in, February of 2011.   And,   yeah, I started doing it full time beginning in 2012.   Yeah. Before that, I, did systems administration in the system software development. So,   yeah, I've I've deployed many cloud in my day.  

[00:03:05] Unknown:

So how did you get introduced to Python?  

[00:03:08] Unknown:

Oh,   I honestly,   I don't really remember.   I mean, I I go back to   to studying computer science in school where I was taught Java and I haven't used Java since then.   And, back in school,   there were there were a lot of people excited about Python, a lot of people excited about Ruby,   and and I was very interested in learning languages.   And   so, I mean, I spent time with with some esoteric. I mean, even went through and learned OCaml.   Spent a lot of time with Ruby, spent,   spent a lot of time in C and etcetera.   And,   after working with Python, I just I just found it a lot more natural of a language to work with, and,   we yeah. I just enjoyed working with Python more. So, I mean, from from an origin perspective, it's more of,   yeah, my my primary argument was generally just preference.  



[00:04:04] Unknown:

It's it's really interesting,   how often OCaml crops up. It's 1 of those languages that even though it's not, like, listed among mainstream   programming languages, per se,   the thread of its, use   winds its way through all kinds of different people and places, and,   it it's seems like a really neat language. I I I have yet to actually take the time to learn it, but,   seems like there's gotta be something there for so many people to have used it and and and cited it as an influence.  

[00:04:36] Unknown:

I I definitely think of OCaml as an influencer,   and I then I'd strongly recommend people to to learn it, not necessarily   to use it,   but but more so to understand,   the flexibility of the functional paradigm   and to understand how that   how how they put object oriented programming directly inside of   a functional paradigm. It's a really fascinating,   fascinating language.  

[00:05:07] Unknown:

So what made you choose Python for   writing SaltStack? And was it just a matter of preference, or were there particular architectural   capacities that it had that made you wanna choose it? And if you were to start from scratch all over again, would you use Python again or would you choose a different language?  

[00:05:24] Unknown:

OCaml's already kind of brought been brought up, so I can't crack a joke and say, Oh, I would use OCaml.   No, I would definitely use Python again and I   think that the choice of using Python   was a good 1 and I look back and   there's there's a lot of things that we've been able to do in Salt because we chose Python.   1 of the main   tenants of salt is its is its plug in system,   and that plug in system is something that   was very easy to write, very easy to make,   make powerful in using Python.   And, because of the way that you can introspect the language, the way that,   you can you can really bend it to your will,   and the very dynamic nature of being able to load things on the fly in a in a reasonable way.  

And so, I mean, that's been that's been a big boon. And then some of the other aspects of Python that have been really beneficial has been,   a lot of the concurrency,   libraries and systems which have more recently come into play. And that's that's 1 of those exciting things where we've been able to   begin implementing more and more of that concurrency aspect inside of Salt to make it   even faster and even more efficient.  

[00:06:44] Unknown:

Yeah. I'll definitely say that having the plug in capabilities has made my life easier on a few occasions, being able   to write your own little module that does exactly what you want and plug it into the overall   execution paradigm without having to dig deep into any of the internals   Definitely makes for a really architecturally   sound   capability.  

[00:07:06] Unknown:

Thank you. Yep. That that was definitely a goal, to make those plug ins as easy as possible.  

[00:07:11] Unknown:

What are some of the challenges you faced maintaining a large and complex,   Python code base like SaltStack?   Does Python make anything easier or harder?  

[00:07:21] Unknown:

That's,   there's there's definitely pitfalls in Python with large code bases, and there's a lot that we've done architecturally   to try and keep that to so that it's less of a complex problem.   So 1 of the 1 of the main things that we've done to alleviate,   just having an overly complex code base in general,   is salts loader and plug in system. And so salt itself, the salt code base is somewhere in the realm of 90% plug ins.   And by somewhere in the realm, I haven't actually done the math you know, I haven't done the calculation on it recently.   And Salt,   has 21 plug in systems right now.  

And,   the fact that we can have these completely   isolated,   plug in subsystems   inside of Solve and having the the code isolation   is has been 1 of the biggest benefits for,   being able to manage,   such a large code base because,   functionally,   the entire capability   of,   every individual component   is reasonably isolated into its own scope,   which makes it really easy to be able to do things like completely rewrite   how, say, we manage a certain container type. I mean, we just completely overhaul   all of our LXC container   systems, and it only meant   effectively editing a couple of files.  

And so that's been a huge benefit.  

[00:08:52] Unknown:

That's great. So something that just occurred to me sort of   first of all, I'm curious what you when you say plug in, separate plug in systems,   I have a funny feeling that there's some meaning behind those words that for those of us who are less,   salt savvy, could you could you explain what what you meant by that as opposed to just straight plug ins?  

[00:09:15] Unknown:

Yeah.   So   a lot of people look at SALT and they see SALT as   a configuration management system. I mean, we generally get lumped in that category. I mean, not generally, almost always.   And the reality is that salt is,   is just is more of a generic management system.   And so   we have many, many different layers of salt which are pluggable. And so when we look at it, you were mentioning before how easy it is to write some functions and plop it in. And so what most people see as our as the primary plug in system is the execution modules.   And that's just running a single salt command and it goes out and fires off that chunk of Python code,   but then the config management system   is all in a different plugin layer called states and those states call   call down to those execution modules so those 2 plugin systems work together.  

And so that's how we get config management is that we take, oh, yeah. We've got command and control, really simple remote execution,   and then we're able to put   and layer a new system on top of it which is configuration management.   But then,   like, the file server back end system installed is a plug in system,   and that's, the component that allows the Saltmaster to directly hook into   a Git repository so that as soon as you commit   your, config management components or your your own plugins   into Git. They're instantly available to solve.   And so since that's a plugin system,   we've got a plugin for Git, a plug in for Mercurial,   SVN, or Azure FS, and Amazon's s 3, etcetera, etcetera.  

And similarly, we've got plugins,   a plugin system for authentication.   We've got we just introduced the new beacon system, which is really cool   because SALT is an entirely event based system.   And so the beacons allows you to watch any arbitrary thing that's going on in your environment and then fire events   so that you can fire an event when specific files are changed, deleted, or created. You could fire events when someone logs in   and then that goes up into the reactor system which is another plug in system.   And so really we Salt is really this combination of all of these plug in systems and realizing that you can build virtually any application that you want to distribute it across an infrastructure   using Salt.  

Sorry. That probably just made made your question to please clarify this into, well, that's confusing and there's lots of bits I didn't know about.  

[00:11:56] Unknown:

Actually, no. I I think I think you did an excellent job because as I said, as I kinda hinted at with my ChefConf reference earlier,   I I work with chef day in day out in my day job. And it struck me as you were explaining it that this must make   adding   not just   new wrinkles, but whole new concepts   easier than a more declarative DSL sort of system because   everything is much more   abstract and modular in its space. Is that right?  

[00:12:28] Unknown:

Very, very much so. Yeah. Because the goal is   is a lot more than setting up and maintaining systems. The goal is   is   keeping track of those systems and managing them in a very holistic way.  

[00:12:42] Unknown:

Yeah. I have to say that the beacon system definitely sounds very intriguing, and I can't wait to have an opportunity to try that out because I've played a little bit with the reactor system, particularly for managing,   doing code deployments from GitHub webhooks. But the beacon system sounds like it's just add a lot more   capability and adaptability to your system. You know, as you mentioned, being able to fire an event when a file gets changed,   there's just no end to the number of different ideas or concepts you could do with that.  

[00:13:13] Unknown:

Yeah. I should say that the excitement around beacons, we we announced them at Saltconf   which was,   yeah, just just 1 month ago.   And   the, yeah, the excitement was overwhelming. I mean, that was the only thing everybody was talking about it seemed like for   for all of Zolgov that they were saying, oh, I can create all sorts of dynamic reactions now based on anything that happens.   And, yeah. I mean, we've got companies. We just added a beacon that every time, So if somebody logs into a minion, every command that they execute on the shell gets fired as an event   so that then they can have an event filter that just   shows them or and tracks   every command executed on every system by that user so that they get this centralized   log of all activity that's happened in their infrastructure.  



[00:14:04] Unknown:

And so, I mean, just all sorts of things like that that people have been really excited about. It's amazing. I bet that would make PCI or SOX compliance a lot a lot simply a lot simpler to manage having the audit trail of everybody who's ever been in any of the systems.  

[00:14:19] Unknown:

Yeah. And that's and that's 1 of the big motivations and we've had a bunch of customers come back and ask us and ask us for it which is why, again, we're very excited to put it in. Great.  

[00:14:29] Unknown:

That follows into our next question of   how do you balance the   desires and needs of the open source community around SaltStack with   the desires and needs of the company behind it.   But do those 2 communities ever come in conflict or has it generally been a synergistic effect so far?  

[00:14:49] Unknown:

What, I mean, what what you're asking is an incredibly tricky balance   and,   and a and a very tricky balance from both ends. And my goal with SaltStack   has been to create a company   that is that fuels the community and is fueled by the community.   And,   I I I talk about this a little bit more in my my keynote   for Solkoff that,   that we should have up on the web here in the next in the next few days.   But the   but the main goal is that we as a company   are able to   gain insight,   and learn from our community and learn how to continue to develop products   to serve the needs of   of others just in general. And we've done a very good job, I believe, of keeping that synergistic.  

And we've done that by making sure that we're not open core.   The plug in system helps a great deal because we are able to,   I mean, there's there's certainly situations where we have proprietary plug ins that we only ship to enterprise customers.   But they're plug ins and they're plug ins that only really make sense to people who are already   buying the, the large system that we're plugging into.   And   the concepts and ideas and innovation that comes out of the community   becomes the fire hose for us developing new features and finding out what our customers really want and what our customers that aren't our customers yet really want.   And it it in a very huge way that synergy has been what drives the business   instead of more of those classical open source business models,   of   going open core   or, trying to rein in what's happening   in the community because the community   can be such a wild wild west.  

I mean, we   we will process sometimes over 200 poll requests in a week.   We're we process more inbound code than anybody else in,   the configuration management communities easily.   And,   yeah, it's it's an incredibly,   active and scary thing to keep on top of.   But it all fuels the company and the company's interests fuel the community.   So so for instance, going back to beacons, beacons were developed to deliver specific,   specific   utilities that customers were asking for.   And we were able to build plug ins that are beacons for those customers   that are specific to their use cases.  

But we were able to then take what that customer wanted   and put that fundamental system in the open, which then drives us to create more things.  

[00:17:42] Unknown:

That's that's awesome. I and and just to speak really briefly on that note, from the Chef community side, I think everybody   in this space is struggling with this a little bit.   And, on the Chef side, I know that they they 1 of the things they've been doing recently is starting up   an RFC process to sort of try to structure some of that community input because the chef community is has   experienced exponential growth, which as we all know can be both a good thing and a bad thing. It's,   so thank you for for filling us in.  

[00:18:15] Unknown:

Yeah. I would I'll definitely agree that the open core model can be very contentious   a lot of times.   It can be beneficial to the project and the company because it does provide a way to   pay the developers who are working on it and drive drive it forward. But when it can be avoided, it's definitely preferable because it definitely bumped up against some circumstances where   there was some functionality   that was in the, you know, proprietary portion of the code base that   wasn't quite,   compelling enough to make me wanna actually jump for the enterprise offering, but just frustrating enough that it turned me away from wanting to the open source portion of the project.   So definitely,   definitely kudos to you guys for being able to avoid the open core model and still be so widely successful as you are.  



[00:19:02] Unknown:

I I appreciate it. And yeah. And it is tricky. The the 1 of the big things that we've discovered is that   when we go in to I mean, we go into a big organization to sell   and some of the benefits of Salt is that it scales well, it works really well in very complex environments   and those companies are more apt to to spend money   on,   on things like SaltStack,   but then the but then the flip side is that much of who we sell to are the groups peripheral   to,   the people who are contributing contributing us   code. So we go into the company and we're able to say to the people who are giving us code,   who are the people around you who would benefit by visibility into Solv?  

And then we can sell those people proprietary software that gives them the visibility into what's going on.   And those sorts of approaches have been very, very,   lucrative for us. It's been   a very interesting approach to be able to say that   instead of trying to bleed money, to to get blood out of a turnip by trying to bleed money out of your community,   we're able to go out and ask the community, who is it that's around you who wants to give us money and we'll go ask them.  

[00:20:17] Unknown:

Yeah. That's a that's a great model and definitely sounds sounds like a very intelligent way to do business with making friends everywhere you go and avoiding making any enemies. So excellent.  

[00:20:27] Unknown:

So far so good.   We'll see if we're still in business in a few years.  

[00:20:35] Unknown:

Yep. I I don't see you guys going away anytime soon. I mean, I I hear a lot about salt, and and I think you guys are experiencing,   you know, maybe not quite meteoric adoption, but but but certainly, I think you're in good shape.   So if you'll indulge me in in, or allow me to indulge in generalities,   developers tend to gravitate towards chef, sysadmins,   and ops folks tend to gravitate towards puppet.   Who might find SaltStack a superior choice and why?  

[00:21:04] Unknown:

So,   traditionally,   sysadmins have been,   more intrigued by SaltStack.   And a lot of the reasons for that is because   Salt does a really good job of solving system administrative problems.   Salt does a really good job of solving problems   that the, that the developer crowd doesn't necessarily see right off the bat.   1 thing that's very 1 thing that's very common that we see is that we'll go that we will see infrastructures   that were   built by developers   and that we see a very large adoption   once those infrastructures get too big to be maintained by the original developers who built them.   And,   and so a lot of it has to do with scale, a lot of it has to do with the fact that   consult just scales up really, really nicely and doesn't need   modification,   outside of, you know, tweaking some configuration parameters to be able to handle a lot of, a lot of scale and to handle really complex   infrastructure problems.  

With that said, we definitely still see,   a fair amount of developer adoption.   And there's a lot that we've been doing recently and a lot that we're working on right now to try and increase that developer adoption.   Because,   most most of our documentation right now is very focused towards system administrators.   It's very   encyclopedic   in nature.   And so we're getting very close. We've got a whole new,   introductory documentation   system that was developed by someone other than me because I'm not good at developing that kind of documentation apparently   that we're gonna be that we're gonna be introducing very soon.  

We put a huge amount of effort into Salt SSH   to to make that whole,   deployment of code and   really seamless, masterless,   management,   really, really slick.   And we're very, very proud of Salt SSH. It is incredibly fast   and gives you   virtually all of the functionality   of remote execution and config management   inside of Salt. I mean, if   with, with this next release that we're finalizing right now,   we're on our second RC,   the salt SSH is to the point where   it can do everything the salt can do without having to without having an agent out there.   And so these systems, we're very very excited about   as we go after that developer market more.  



[00:23:37] Unknown:

So the the Salt SSH   aspect sounds really neat. Is that perhaps an answer without mentioning names   to some other configuration management systems that are gaining ground because of super easy adoption?   You know, like, because they people can start out with just simple orchestration   and add very simple   configuration   in. I'm thinking of the thing that could be used in a blacksmith shop.  

[00:24:05] Unknown:

We learned we we've learned a lot from our competitors   and,   definitely watch what they are doing.   And I should say that I am I'm very grateful to them from what we've been able to learn.   And,   so, yes, there is definitely some answer there.   And so all this to say historically is, I mean, it's a it's a really interesting thing because   originally, we wrote it to extract the transport system out and just kind of clean up the code   and, very much so as a challenge to see, hey, could we actually do this?   And then we kind of we kind of stepped away from it for maybe 6 months or so   and said, hey. We accomplished this. We've got a lot of other big things to deal with to handle our big infrastructure customers   and our big infrastructure users.  

And,   we more recently came back because we saw more and more demand around wanting Salt SSH.   And so,   most most of our recent work has definitely been an answer to our competitors.   And and we really look at that and hope that, getting out, better introductory documentation,   helping people understand that from a high level, we still definitely believe that Salt is the easiest to use   configuration management system out there.   And that unlike a lot of other systems out there,   we started by solving the hard problems at scale   and that we wanna be able to to deliver something that is easy to get started with, easy to roll out, all those things that you're saying,   but that will grow with the person who's deploying it because we've seen so   so commonly   these infrastructures   get up to a few 100 servers and then they come over and they say, oh, we need to scale now and these systems are so slow and what's taking us   hours to do takes us minutes and seconds to do a solve.  



[00:26:04] Unknown:

That's that's awesome. And I think the marketplace has shown,   has spoken very clearly that   a really easy ramp up and adoption curve is super important and and that   in a way, it's kind of less important that   the best configuration management system win, but that the easiest win,   at least as far as the market is concerned. So I know Chef's been doing it a lot in that sphere as well. I'm really glad to hear you folks are are recognizing the importance of that and, and tooling appropriately.  

[00:26:37] Unknown:

Yeah. Yeah. And I, yeah. And we've we've really gotta get out and market more of what Salt SSH can do now.  

[00:26:44] Unknown:

Alright.   So 1 thing that Salt   has that's both a benefit and sometimes potentially a detriment is the fact that it   doesn't really impose any particular rigid structure   on how you make use of it. So I'm just wondering   what some of the complications   you've seen arise out of that of, you know, not having a particular prescribed approach to   using it,   and also, you know, some of the major benefits that you've seen as a result of that as well?  

[00:27:16] Unknown:

So 1 of the 1 of the things that I think is interesting about Salt is that I tried to make it so that the config management system would be easy to learn,   easy to transfer from person to person, tried to make it so that the the systems are really easy to interface with.   But   as salt is grown   exactly what you're saying is what's happened is that   is the salt itself is fundamentally built to be flexible enough to solve any problem that you can throw at it.   And the problem that we're into is that is that that could be very overwhelming for people   that we that we basically say, hey. Have a ton of rope.  

You could build you could build whatever you wanna build.   And I mean, the benefit to that is   very, very straightforward when we look at some of the things that people have built. I mean I mean, SALT is being used literally to manage cruise ships.   I mean, we talk about, like, Internet of things.   SALT is used to manage cruise ships. It's used to manage slot machines. It's used to manage   and deploy code on weather stations in South Africa.   And,   that loose coupling is what enables   that level of integration,   and that loose coupling is what enables us to, you know, do whatever needs to be done.  

And it's and and it horrifies people when they come to me and they say, well, Tom, what's the, you know, what's the right way to use salt? And I say,   the right way to use salt is to use it in a way that solves your problems.   And, yeah,   it's it's a very different paradigm   because a lot of people wanna be handed something that says, here's your rule book so that you don't get into trouble.   And, we've we've worked really hard to structure salt so that it's it's harder to get into trouble.   But it does it does lend a bit of a challenge when it comes especially to the configuration   management end of salt   that,   that people want that structure in place.  

And as such, we have developed,   best practices   guides around how to build the configuration management   in a way that is more maintainable and is more transferable.   And putting a lot of that emphasis on, you know, making really generic   salt states and   using pillar to,   to steer what's happening,   which is really the core idea of pillar.   But it but it definitely has been a challenge.   And people are more apt to use a tool out of the box, I will say,   that gives you those rigid guidelines.   But   I feel confident that we will continue to grow. And   like you were saying, we don't we've never grown at a meteoric pace.  

We've grown at a steady and a strong pace.   And I think that that has to do with the fact that   the salt is something which can adapt itself to any scenario.   And we're not out there drumming up the hype.   We're not out there saying, oh, 0, look at the, you know, look at the shiny bits. We're out there saying, we will solve your real problems   and help, you know, and make make those problems   go away   in a real way instead of, you know, trying to sell the snake oil of the moment.  

[00:30:41] Unknown:

That's that's excellent, and I I'm glad that you folks are are putting some focus into   giving people some some guidelines, because I think, you know, that the other product that we were speaking about, previously,   I think that's sort of 1 of the ways that they've   gained such a tremendous amount of of of adoption of late   is by giving people, like, here, look, you can do this very easily. And it's a very small thing. Right? I mean, the truth is most of what people start with could be achieved very easily with other tools,   but it's like people love that. And if you sort of make it really clear to them, make it very concrete, to   offer   them,   the   to offer them,   the the ability to scale and the ability to build really complex infrastructures,   you know, even   by comparison,   much more so than some other of its competitors,   I I think that's great. I'm glad I'm glad to see that.  



[00:31:40] Unknown:

Yeah. I will say that,   to your point, Chris, you know, there are certain different tools that you can use for doing number of the various things that SaltStack can do as well, but that's the problem right there is that they are different tools, whereas Salt,   you can use 1 tool to do all of it. That that's been my experience.   At a previous company I was working at, we act had actually just had a similar issue where we were using 1 tool for orchestration and a different tool for configuration management.   Trying to figure out how those 2 interacted   was sometimes problematic,   and so we actually ended up, starting to move to using SaltStack so that we could just consolidate all of those things and just have everything in 1 place so you didn't always have to jump back and forth between 2 different code bases and 2 completely different tools and programming languages and paradigms to try and understand  

[00:32:28] Unknown:

what actually happens when you issue a certain command. Well and and and it's reassuring to hear what you're saying about it because we're we're in the process of trying to develop exactly what you're what you're describing along the lines of   giving people short videos and tutorials around   here's how you solve small problem x in solve.   And,   also emphasizing those around, here's how you solve a problem that you thought wasn't small but here's how you do it in 2 minutes with solve.  

[00:32:58] Unknown:

That's that's very that sorry. That's very smart. Right? Because if they if they feel like it's big and they're shown, here's how to make it really easy, then that's gonna be tremendously satisfying for people.  

[00:33:10] Unknown:

Yeah. I can't I can't tell you how many people have come back and done things like,   like set up   completely automatic   code deployment or canary environments with salt that nobody I mean, you don't even have to touch it. Somebody pushes code   and the canary environment gets deployed for them and   they're done. And people are amazed at the fact that they do, that you could do that,   in less than 10 lines of, I shouldn't, I shouldn't quote it.   It depends on the tool.   But in in minutes, you can set up these really complicated fully autonomic systems.   And so yeah, we're we're definitely gonna be coming out with a lot more,   instruction and documentation   and videos around how to accomplish those things.  



[00:33:59] Unknown:

Wow. Thomas,   if you wouldn't mind,   especially if it's if it's available on the web, after the fact, it'd be really cool if you could link that that,   10 line   canary environment   deploy or thing   into the show notes or hand it to us or whatever. If it's if it's assuming it's publicly available, because I think that would be a great,   agreed opportunity for people to get a sense of how much salt can do.  

[00:34:24] Unknown:

No. I'll definitely do that.   Yeah. We we need to get that 1 out there. We set up,   we've set this scenario up a lot. It's actually the core idea came from   Wikipedia.   And so,   yeah, because because Wikipedia, as you probably know, uses Puppet for their configuration management,   but they use, but they use Salt for their orchestration and deployment.   And many of their and much of the configuration   management that they use in newer deployments is pure Salt.   And that they they were incredibly early adopters   and they were the first ones to come back and say, hey, you know, I can   use SALT's peer system to pull this off. And after looking at how they developed it,   we developed,   we developed the event reactor system, which   which you've already mentioned and developed that system   in such a way to make it to make the more complicated scenario that they built out something that was much easier, much more reproducible.  

And of course, you could do flipping anything with the event reactor system now.  

[00:35:31] Unknown:

Very cool. So following on to to that that question, what are some common problems that you've seen people struggle with when using SaltStack?  

[00:35:41] Unknown:

Getting back to their day job.   Actually, in in all honesty, that is a fairly common problem that we see is that   is that people will find salt and they will go down the rabbit hole of just creating some of the most fantastic things.   And then they kind of poke their head up and go, oh, everyone else is the the people that come and they just beg to work for us and say, I wanna build these fantastic things all day long.   So for instance, 1 of my favorites is that at Hewlett Packard, they developed a   a complete app   container deployment system around Salt. They put it together in 3 weeks.  

It is a stable app container deployment system.   They tossed a lightweight GUI on top of it and so people inside people inside of Hewlett Packard for internal   development purposes   are able to   just give   a container definition and it's instantly deployed and it's all salt under the hood   and salt calling out to LXC   and to the extent where their users can use our Docker to LXC conversion   components   and just develop everything in Docker and plop it into Salt and it deploys LXC containers.   And and that I'm trying to I'm I'm actually kind of being a punk and turning your question of what are the problems with salt into a salt's fantastic   answer.  

Usually some of the things that people have hiccups with is that they might, you know, tread onto territory that's really, really new and run into some some bugs that we hadn't hit yet,   or more of it is just kind of getting lost in the functionality.   It's it's that people will pick up SALT and say, hey, I'm gonna try out salt for config management.   And they don't they don't realize the salt is more of this total management   system   and, they start worrying too much about components that they don't need to worry about. So, I mean, I guess a lot of my answer there would be that if so if you're using salt   and you're worried going, oh, well, I don't understand how all these different systems work. Even in the config management system, if you're using Salt and you go, oh,   I'm intimidated by prereq and all of these,   declarative components, which looks scary.  

Don't worry about it. You don't need to worry about those things.   Just know that they're there.  

[00:38:13] Unknown:

It's like the old adage, how do you eat an elephant   1 bite at a time. Right?  

[00:38:18] Unknown:

Right. Right. We we generally compare it to, like, a SWAT team,   you know, or police a police department. You go in and your handgun will take care of the vast majority of situations, but you know   that there's an assault rifle back there or tear gas or whatever you need to solve any problem you might run into.   And that's kind of the benefit of salt that no matter what you run into, we will we we have a solution for you.  

[00:38:45] Unknown:

That's great. And and with regards to your getting back to their day job problem, well, if if they really wanna do this kind of thing all the time, I mean, there are so many companies hiring for   infrastructure as code or or DevOps or whatever you wanna call it. I don't think they'd have to look very hard if they wanted to change, change disciplines and and do this stuff for a living. Now I don't know I don't know how many are advertising   explicitly for salt stack, but a lot of them are so desperate that if you found someone who said, I know salt and I can make it do wonders and and and, you know, wash your dishes and get your paper, I'm sure a lot of companies would be much more,   open to using it given how desperately they're in need of the help.  



[00:39:25] Unknown:

I gotta say 1 of my 1 of my favorite   stories, or I should say 1 of my favorite kind of stories is   is to hear about how many people   have been introduced to solve who   had minimal systems knowledge,   minimal programming knowledge,   and we're able to dramatically improve their career over a very short period of time.   I mean, 1 of my favorites was somebody in our community who was working in a in a call center for a hotel chain   that never done system administration   before   and heard about assault through just 1 of his friends said, hey. Salt's cool. Look at it. So we popped into IRC   and studied our documentation   and started helping people in IRC.  

8 months later, the guy's making 6 figures   at a   at a major tech company in Seattle   to pull up SALT.   You know, and it's like, oh, well, you just got a a, like a 4 x pay increase, buddy.   But we have a ton of stories like that.   Yeah. And and it's very it's it makes it very rewarding  

[00:40:34] Unknown:

to be able to feel like you are actually helping people on a on a more personal level instead of just saying, hey, your servers work better. Yeah. I'll definitely say that 1 of the troubles that I've seen a couple times when I've been using solve is running into that analysis paralysis of, okay, the there's this particular problem I'm trying to solve, and now I see, okay, there are these 5 different ways I can use solve to solve it. So which 1 is going to be optimal for my particular   situation.   But sometimes you just kinda have to pick 1 and run with it. And then if it doesn't work, eventually, then you can go back to refactor that, which it will say SALT makes relatively easy because there is generally so little actual coding that you need to do to tie the different components together.  

[00:41:15] Unknown:

Well and and, yeah, the analysis paralysis, I mean, we will have,   I mean, I'll have all of some of our,   our implementation   and support   people come up to me and say, oh, Tom. I don't know how to solve this problem over here.   And and I, you know, I have no idea what's going on. I'll sit down and talk to him for 2 minutes and say, oh,   don't you know about,   using an external pillar?   And they go, oh, that's an external pillar.   Or yeah. Oh, and I'm like, oh, well, you're gonna write 15 lines of Python code and hook into their existing CMDB and you're done.   And but, yes, we we see it all the time. It's that there's so much there that   either you don't know about or if you do know about it's kind of   this feeling of, I've gotta figure out what these 8 things do before I know which 1 to use.  

And it's it's really just a challenge of building a truly flexible system to be able to do that, and and I'm hoping that   us putting these videos out that are able to say, x y z, give   give people these concrete examples   of   this is this is a good way to use   I'm trying to think of something more esoteric inside of salt, but I've already used up so many of them. Oh, but, like, this is a good way to use returners   for auditing, or this is a good way you know, this is a good use case for building an engine inside of Salt   and using that. And,   hopefully, we can get those we can get those rolling soon. We just we we we just finished up this release, and I imagine you guys, are familiar with how much of a pain it is to to finish up major releases.  

Oh, yes.   Oh,   and we just finished the conference. Conferences are murder.   I mean,   they're   conferences are so fantastic, so much fun, but putting on a conference   is just the most painful thing in the world.  

[00:43:15] Unknown:

Well, I think I'm sure it's it's it's not unlike doing a major release. Right? From the outside, people are like, well, come on already, like, where is it, you know? Shouldn't you have been done 2 weeks ago? But there you are laboring in the trenches because it becomes such a fractal problem of achieving that last, you know, 2 feet to cross the finish line. It's like, oh, we just have these really critical problems we have to solve, and then we can do it. And and, it seems like putting on a conference is is very similar. From the outside, it seems like, so what's the big deal? You rent a venue, you get some speakers, it's easy. But there's this   incredibly   fractal amount of work behind the scenes to arrange all the logistics to make it happen.  



[00:43:53] Unknown:

Oh, tell me about it. I mean, for our first conference, I had to write,   call what was it? I had to write about a 150 pages of training   manual content,   in, like, a week and a half before that conference happened.   And it's just just insane doing that stuff. So yeah.  

[00:44:19] Unknown:

So what are some of the most surprising or interesting things that you've seen people do with SaltStack? Things that you just never would have even imagined possible with with, when you first wrote it.  

[00:44:30] Unknown:

Oh,   there's there's a lot.   We   we get used,   a great deal for,   probably the first 1 that surprised me when I found out about Salt being used to manage medical devices.   That,   that they were using salt on,   on, embedded devices   that were being put throughout a hospital.   I don't even remember what they're monitoring but I mean it was monitoring something   relative to the patients   and then using salt as this data aggregator.   And that was back in, that was back in 2012 when SALT was just a year old.   And,   we look at it now   and, our   current play and a lot of what we're working on for security mitigation   has   kinda surprised me.  

I can't tell you how proud I am of all the tweets that come out after heart bleed and shell shock that were people saying hey,   I mitigated shell shock in 4 minutes with salt stack over 10,000   servers.   Wow.   Oh, I love those. And I come back and I say, 4 minutes? What took it so long? I gotta   and they say, oh, don't worry. Salt had actually done its work in a minute and a half, but the other time was testing.  

[00:46:00] Unknown:

Yep. I definitely saw a number of those tweets come out after those events as well. It's definitely   gratifying to see that.  

[00:46:07] Unknown:

Yeah. It's very reassuring.   And,   but probably 1 1 that I'm that I'm really proud of most recently was,   was that a nonprofit   set up a bunch of weather monitoring stations using raspberry pies   in,   in South Africa.   And these stations are used to give,   to give early alerts   to to locals who otherwise don't have access   to this kind of information   or don't have access to,   to be scanning for this information.   And so SALT is used to set up all of these weather stations and then the SALT event bus   is used to channel information   back to a central hub about what's going on.  

And and that whole   Internet of things   component   is   just something I'm really excited about. That we've got companies that are starting to use SALT   for, for home automation purposes   inside of, yeah, inside of just arbitrary home automation devices.   And they love it. They come back and they say this is great because I've got a home automation device   and everything is going through this standard interface that salt just gives me.   And so, yeah, that that stuff is a lot more esoteric   than,   oh, we manage a big data center, which we do lots of that.  

[00:47:33] Unknown:

So when you you talk about the Internet of Things in these weather stations and things like that, just for the purposes of those of us who are who are less salt savvy,   would that be something like a minion running on on that Raspberry Pi?  

[00:47:48] Unknown:

Yeah. Yeah. So so   people will people will put a minion on the Raspberry   Pi and then just have a master running back,   either at an office somewhere or in a central, you know, some central location.   And all of those minions are just reporting back into that master.   And and that's it's actually interesting too the different ways I've seen people do it because   a normal, you know, a classical salt deployment is that you've got this you've got the 0MQ,   maintaining this cons this, TCP connection back to the master.   But we recently introduced rate which is a UDP protocol   and we've got the ability to and we recently   not not too recently, it's maybe a year and a half old, the the scheduler in salt. The recent thing about salt's,   schedulers that we added, we just added tons of,   additions onto its capabilities   so that your scheduling can be incredibly granular now.  

But a lot of people will just fire up that minion   and say I don't care about the constant connection   or the TCP connection isn't gonna do the right kind of traversal but I can I can route it with the u with Salt UDP protocol   or I can just set the scheduler to have that minion act as this detached agent   that is either pinging a master on a regular basis   or we just take the master out of the equation altogether   and all of those minions are just using Salt's returner interface   face and the returner interface is Salt's,   plugin interface that allows you to connect to virtually any database?  

And so they just use Salt's returner interface   to run commands on a schedule and just have that go to some arbitrary database.   And,   and it's that flexibility, that fundamental   decoupling   that is really what allows SALT to work in so in such disparate environments.  

[00:49:45] Unknown:

That's very cool. And I and I really do think that that is a distinguishing   feature of salt   versus almost any of the other configuration.   Well, I can't call it configuration management system as you pointed out. That's that's a misnomer.   Well, I guess   that's a really distinguishing feature compared to other configuration   management systems of which Salt is not 1.   On the market,   it it really sort   of, like you say, lend it to be used in in areas,   or environments   where traditional Centimeters   probably   doesn't really can't really offer that much or   bringing it to bear or bringing them to bear rather might be a little bit obtuse or or hard to get around.  

Very cool.  

[00:50:30] Unknown:

Yeah. 1 1 thing that I've seen a few different implementations of that I thought was very interesting was actually using Salt as the monitoring and alerting platform for an infrastructure   where,   they had a scheduler set up to have the minions   send back the, you know, CPU statistics, memory statistics back to a particular returner back end, whether it be a graphite database or a relational database or anything that you can see some on Elasticsearch. And then also using that same capabilities to actually generate alerts from that data,   is definitely pretty cool.  



[00:51:03] Unknown:

Yeah. Yeah. And that's and that's 1 of the big pushes that we're working on right now because,   because we wanna make sure that we're able to   really satisfy all those needs   and make it much more streamlined to use SALT for your full stack of monitoring.   And really it's why not? I mean, we've already got this execution library of modules that can already do all the monitoring for you.   Great.  

[00:51:30] Unknown:

Yeah. That that is that is great.  

[00:51:32] Unknown:

Yeah. 1 of the things that people say about Salt's config management is they say Salt is a config management system in the same way that Google is a search engine.  

[00:51:41] Unknown:

Or perhaps, emacs is an editor. Right?   So,   just to sort of,   I guess, as an as an homage to 1 of my favorite,   technical podcasts that, and partially inspired me to want to do this 1, we have,   what we call the picks,   that we'd like to do. And we'd like you to pick, you know, a few things. They can be technical or non technical, just things that you like. It can be   software technology. It can be beer or a board game, something in a smartphone   app, you know, whatever it is, just something you you think that's cool and you'd like to tell people about.  

[00:52:23] Unknown:

Oh, something that I think is cool that I would like to tell people about.   I'm a big fan   of flow based programming.   I think that flow based programming paradigm offers a   completely different approach   to how you think about computing and especially distributed,   distributed systems.   And we've been playing around with a lot trying to figure out a good way   to implement flow based programming inside of salt in such a way that that optimizes things.   So, I mean, that's that's definitely something that I would encourage people to look into.   I'm trying to think of something that's nontechnical but I'm terrible because my entire life has been so incredibly obsessed   with,   with salt recently   that, you know, I I I don't get out much.  



[00:53:15] Unknown:

That's quite alright. I can definitely relate to that, but   go go for the technical.  

[00:53:21] Unknown:

Yeah.  

[00:53:23] Unknown:

Yeah. So can you give us an example of of a flow based programming system?  

[00:53:29] Unknown:

Yeah. So   Salt,   1 1 thing that's that, that we dug up to support Salt is something called IOflow.   So I used to work in the US intelligence community as a contractor,   and so I can neither confirm nor deny many things.   But I became aware of this flow programming,   platform that they were using to manage robotics called Ioflow.   And, I knew the developer   and, knew the right contact contacts.   And, well, we call it Ioflow. Internally,   it was something else.   And, I was able to to coax it out into the open   once it was renamed and scrubbed, you know. Right. Yep.   And,   and the ioflow project   is an incredibly   powerful   flow programming   system.  

It's pure Python   and,   I'm not gonna say that it's easy to learn how it works. It's kinda complicated because, I mean, it was written in the dark for,   like, autonomous   vehicles and submarines.   But when it comes to building autonomous systems,   it's it's incredibly useful. I mean, it takes, again, it takes a while to get your head around. All the documentation   is obtuse and very academic,   but it's just this incredibly cool system.   And the whole idea behind flow programming is that   you you've got this data bus and so   all of the data for your application is on this data bus. You don't make objects with isolated data.  

And then you've got these black boxes. Ioflow calls them behaviors   and then those behaviors access the data bus.   And so the behaviors are restricted to only be able to get certain data off of the data bus and so then they can read and write certain data off of the data bus   and then everything is all of these behaviors are just on this schedule checking the bus to say,   did something change? Oh, that thing changed so this behavior should run.   And it's just a fascinating programming paradigm and again I think a very, very, very powerful 1. Nioflo is a   fantastic example of how it works. I'd really like to,   to extend ioflow or build a system like ioflow,   that better abstracted   the concepts of flow based programming.  

But, I mean, 1 thing at a time.  

[00:56:04] Unknown:

Absolutely.   Is Ioflow open source? I see Googling it. I see   a Microsoft Research Publications   paper on it.  

[00:56:13] Unknown:

Oh, you're probably spelling it wrong. It's I o f l o. There's no w.   Ah.   And it's Apache licensed.  

[00:56:22] Unknown:

Oh, perfect. Thank you very much. That helps a lot.   There we go. I found it. It's on GitHub. Super cool. Thank you. We'll put that in the show notes. That sounds really neat.  

[00:56:31] Unknown:

Yeah. Definitely cool stuff.  

[00:56:35] Unknown:

That's great. Don't feel any pressure. You gave us a really good 1 there. Do you have any other picks you'd like to cite?  

[00:56:42] Unknown:

Oh,   no. No. I don't know.   Imagine Dragons are awesome, but everybody already knows that.  

[00:56:49] Unknown:

Hey, man. That works. It's you know, music is 1 of those amazing things that that there there is literally,   you know, there's something for everyone. So I I like Imagine Dragons too, and and who knows? You know, this this podcast reach could reach someone in outer Slavovia who's never heard of the Imagine Dragons. And now they'll be prompted to go on Spotify and give them a listen. So No. You just reached 1 because I've never heard of them. Well, so there you go. There you go.   Oh,  

[00:57:16] Unknown:

man. They they wail. You gotta check those guys out. Yeah. Well, as soon as we're off this call, go check it out.   Very cool.  

[00:57:23] Unknown:

So So I guess I'll move on to my picks.   And the first pick that I will pick is   a beer because I like good craft beer. When I was in California, 1 of the things I love about about craft beer is that as I, you know, as I get to travel,   I really enjoy trying the craft beers that are only available in that region because some of it doesn't travel particularly well.   And I had the opportunity to try   a pint of stone Imperial Russian Stout,   and it was really delicious.   I I really love Stone Brewery.   I find that their beers   really sort of like hit the nail on the head with craft beer. Sometimes,   you know, with with things like a stout or or there are few other varieties like this. There's so much complexity there that it's very easy for a brewer that's not particularly expert   to have their beer come out kind of 1 note.  

And, this 1 is balanced perfectly. It's delicious malty goodness. It's it's great stuff. If you like dark beer,   or even if you,   think you don't like beer at all, but have only been fed to things like Budweiser, not to say there's anything wrong with Budweiser,   or or very light, hoppy beers, you might give it a shot.   The second pick that I have is actually a blog post   by   Arun   Ravindran.   I'm sure I mangled his name horribly being an insular American who can't pronounce foreign words,   but it's called Python 1 liner games.   I'll put the link in the show notes. And what I find to be really cool about this post, these are not games like,   I don't know, you know,   metal metal gear solid or anything like that. These are like computer games,   like those of us who are old and crusty   used to play in the days when we used to type them in in basic, you know, things like that. Things like,   character generated mazes   and snake games and, like, really just simplistic things. But they're 1 liners, and thus, I think they're very cool. I I'm a big fan of whimsy in programming and I think that, that we need to get it any anyway we can. So I really enjoyed that post.  

And   the third thing that I wanna pick is I wanna pick the Boston Python meet up. I went for the first time a couple of weeks ago,   and I really really enjoyed it. It was the the the quality of the of the content being presented was top notch, and I met some really cool people. And in general, I mean, if if you're not involved in meetups near you, and chances are there is 1 near you, unless you also live in out of suburbia.   It's it's definitely worth taking the time to drop by and say hello. You you you chances are you won't regret it.   That's it for me. Great.  



[01:00:08] Unknown:

So for my first pick, I'm going to pick a, somewhat self serving 1 that's,   rather topical. So I actually   recently wrote a couple of blog posts about my experience in using SaltStack to   move a friend and   client   from Heroku onto AWS.   So I'll put links to those 2 posts in the show notes.   For my next pick, I'd like to choose a   there's a tool called CaveQ   that is a clipboard manager that I use on the KDE desktop on Linux. I believe it's cross platform to at least OSX and, I believe, Windows,   but it is an amazing clipboard manager. It lets you maintain clipboard history. You can edit it. You can   categorize it into separate tabs.  

So, for instance, I use EMAX as my main editor.   And whenever you highlight a line in emacs or, you know, copy it into an emacs buffer, it also shows up in your clipboard. So it was just spamming my main clipboard   and just pushing everything relevant back off of the, list. So I I was able to set up a filter so that anything coming from my emacs program went into its own tab, so it was still accessible and searchable,   but it wasn't,   it it wasn't polluting my main   clipboard history. So it's a phenomenal phenomenal system,   definitely worth checking out.  

And I'll also choose the Pelican static site generator.   That's actually what I use   to write and run my blog. It's a Python based static site generator, has a really good plug in system, has a really good repository of themes that make it really easy to get up and running with a decent looking website,   has a lot of different capabilities. You can use it for a regular generating regular websites.   It's primarily used for blogging, but   phenomenal,   very popular project. Definitely worth checking out.   And last 1 I'll pick today is a website called Noisli,   n o I s l I,   and it is   a white noise generator.  

So you go there, and there are a number of different   sounds that you can choose, and you can adjust the volume. So for instance, you could say, I wanna have a little bit of wind, a little bit of crackling fire, and maybe a thunderstorm in the background. You could just mix and match from all those different sounds   to make a really pleasant,   ambient sound for you to listen to when you're really trying to focus.   I found sometimes when I was listening to music while I program, you know, it could be great to be motivating, but sometimes   they can just be a little bit too distracting, and I found Noisli to be perfect for those situations.  



[01:02:42] Unknown:

Very cool.  

[01:02:44] Unknown:

That is all I have for PIX.  

[01:02:46] Unknown:

That's that's awesome.   The the noisily thing sounds really neat. I mean, there there are certainly I I have a couple of old CDs that I've been using for, like, 20 years,   called environments. And there's there's a couple in particular, like the ocean waves and and rain. And I I have yet to find any kind of digital   thing that comes as close to,   filling that niche for me as these recordings. I'll have to give Noisli a try and see if it,   see if it passes muster.  

[01:03:16] Unknown:

So   that does it for this show.   Thomas, is there anything that you'd like to plug or any appearances that you're gonna be having anytime soon   that you'd like to let our listeners know about?  

[01:03:31] Unknown:

Well, I mean,   there I I would like to you you invited me to plug something, so I will.   And you were talking about PIX.   Another project that I put together is something called libnagle,   which is a,   which is,   Python bindings to the libsodium library.   And they've been abstracted in such a way to make cryptography   incredibly safe and easy and   incredibly easy to use.   So,   you know, if you ever need   super strong cryptography, take a look at libnacl.   Yeah, libnacl.   But,   but, yeah, we've got, I mean, we're gonna be we're gonna be a PyCon.   We're over at   the, what is it, the Microsoft Build Conference.  

Yeah. Salt's all over the place.  

[01:04:29] Unknown:

So, yeah, keep an eye open on what we're doing. Okay. And for anybody who'd like to follow you in the SALT project, where can they find you?  

[01:04:38] Unknown:

Well, I am thatch45   on GitHub.   I am thatch45  

[01:04:43] Unknown:

on  

[01:04:46] Unknown:

Twitter. I don't I don't tweet very much. I keep trying to tweet more but I'm a bad tweeter.   But, yeah, definitely follow the Salt Project which is at GitHub, Salt Stack, Salt Stack slash Salt   and we've got a bunch of other really cool projects in there.   And,   yeah, join the community. We're we love we love merging code. We,   and we work really hard to have a nurturing and a very positive community that helps people   learn how to improve   which has worked out really well for us. So,   yeah, definitely   look us up.  

[01:05:23] Unknown:

That's great. I I just have to ask, sir, what, what's the 45 in THATCH 45?  

[01:05:32] Unknown:

So   there's there's a bit of a story.   45 is my lucky number because when I was 10,   I was playing Legend of Zelda on the 8 bit Nintendo,   and I closed my eyes and named my Zelda character. And he ended up and Link ended up being named 45.  

[01:05:55] Unknown:

Oh, that's awesome.  

[01:05:56] Unknown:

And it was the best game of Zelda that I'd ever played.   I got I got through, like, the 5th labyrinth without dying.   And so I just always used 45.  

[01:06:07] Unknown:

That's awesome. Thank you. I'm very glad I asked.  

[01:06:13] Unknown:

Well, thank you very much for listening. Thomas, thank you for joining us, and we are looking forward to bringing you another great episode as soon as possible.