Infrastructure

Kinto with Alexis Metaireau and Mathieu Leplatre - Episode 86

December 4, 2016

Summary

Are you looking for a backend as a service offering where you have full control of your data? Look no further than Kinto! This week Alexis Metaireau and Mathieu Leplatre share the story of how Kinto was created, how it works under the covers, and some of the ways that it is being used at Mozilla and around the web.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.
  • When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for running your awesome app.
  • You’ll want to make sure that your users don’t have to put up with bugs, so you should use Rollbar for tracking and aggregating your application errors to find and fix the bugs in your application before your users notice they exist. Use the link rollbar.com/podcastinit to get 90 days and 300,000 errors for free on their bootstrap plan.
  • Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • Your host as usual is Tobias Macey and today I’m interviewing Alexis Metaireau and Mathieu Leplatre about Kinto

Interview with Alexis and Mathieu

  • Introductions
  • How did you get introduced to Python?
  • What is Kinto and how did it get started?
  • What does the internal architecture of Kinto look like?
  • Given that the primary data format being stored is JSON, why did you choose PostGreSQL as your storage backend instead of a NoSQL document database such as CouchDB?
  • Synchronization of transactions from multiple users, including offline first support, is a difficult problem. How have you approached that in Kinto and what are some of the alternate solutions that were considered?
  • Designing usable APIs is a complicated subject. What features did you prioritize while creating the interfaces to Kinto?
  • What are some of the most innovative uses of Kinto that you have seen?
  • What are some of the biggest challenges that you have faced while building Kinto?
  • What do you have planned for the future of Kinto?

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Python at Zalando - Episode 74

September 10, 2016

Summary

Open source has proven its value in many ways over the years. In many companies that value is purely in terms of consuming available projects and platforms. In this episode Zalando describes their recent move to creating and releasing a number of their internal projects as open source and how that has benefited their business. We also discussed how they are leveraging Python and a couple of the libraries that they have published.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • Rollbar is also sponsoring us this week. Rollbar is a service for tracking and aggregating your application errors so that you can find and fix the bugs in your application before your users notice they exist. Use the link rollbar.com/podcastinit to get 90 days and 300,000 errors for free on their bootstrap plan.
  • Hired has also returned as a sponsor this week. If you’re looking for a job as a developer or designer then Hired will bring the opportunities to you. Sign up at hired.com/podcastinit to double your signing bonus.
  • Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we’re interviewing Jie Bao and João Santos about their use of Python at Zalando

Interview with Zalando

  • Introductions
  • How did you get introduced to Python? – Tobias
  • Can you start by telling us a bit about what Zalando does and some of the technologies that you use? – Tobias
  • What role does Python play in your environment? – Tobias
  • Is the use of Python for a particular project governed by any particular operational guidelines or is it largely a matter of developer choice? – Tobias
  • Given that you have such a variety of platforms to support, how do you architect your systems to keep them easy to maintain and reason about? – Tobias
  • One of the projects that you have open sourced is Connexion. Can you explain a bit about what that is and what it is used for at Zalando? – Tobias
  • What made you choose to standardize on Swagger/OpenAPI vs RAML or some of the other API standards? – Tobias
  • Did Connexion start its life as open source or was it extracted from another project? – Tobias
  • ExpAn is another one of your projects that is written in Python. What do you use that for? – Tobias
  • Can you describe the internal implementation of ExpAn and what it takes to get it set up? – Tobias
  • Given the potential complexity of and the need for statistical significance in the data for proper A/B testing, how did you design ExpAn to satisfy those requirements? – Tobias
  • Given the laws in Germany around digital privacy, were there any special considerations that needed to be made in the collection strategy for the data that gets used in ExpAn? – Tobias

Keep In Touch

Picks

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Test Engineering with Cris Medina - Episode 68

July 30, 2016

Summary

We all know that testing is an important part of software and systems development. The problem is that as our systems and applications grow, the amount of testing necessary increases at an exponential rate. Cris Medina joins us this week to talk about some of the problems and approaches associated with testing these complex systems and some of the ways that Python can help.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry’s real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at getsentry.com
  • Hired has also returned as a sponsor this week. If you’re looking for a job as a developer or designer then Hired will bring the opportunities to you. Sign up at hired.com/podcastinit to double your signing bonus.
  • The O’Reilly Velocity conference is coming to New York this September and we have a free ticket to give away. If you would like the chance to win it then just sign up for our newsletter at pythonpodcast.com
  • To help other people find the show you can leave a review on iTunes, and tell your friends and co-workers
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we’re interviewing Cris Medina about test engineering for large and complex systems.

Interview with Cris Medina

  • Introductions
  • How did you get introduced to Python? – Chris
  • To get us started can you share your definition of test engineering and how it differs from the types of testing that your average developer is used to? – Tobias
  • What are some common industries or situations where this kind of test engineering becomes necessary? – Tobias
  • How and where does Python fit into the kind of testing that becomes necessary when dealing with these complex systems? – Tobias
  • How do you determine which areas of a system to test and how can Python help in that discovery process? – Tobias
  • What are some of your favorite tools and libraries for this kind of work? – Tobias
  • What are some of the areas where the existing Python tooling falls short? – Tobias
  • Given the breadth of concerns that are encompassed with testing the various components of these large systems, what are some ways that a test engineer can get a high-level view of the overall state? – Tobias
    • How can that information be distilled for presentation to other areas of the business? – Tobias
    • Could that information be used to provide a compelling business case for the resources required to test properly? – Chris
  • Given the low-level nature of this kind of work I imagine that proper visibility of the work being done can be difficult. How do you make sure that management can properly see and appreciate your efforts? – Tobias

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Sentry with David Cramer - Episode 61

June 12, 2016

Visit our site to listen to past episodes, support the show, join our community, and sign up for our mailing list.

Summary

As developers we all have to deal with bugs sometimes, but we don’t have to make our users deal with them too. Sentry is a project that automatically detects errors in your applications and surfaces the necessary information to help you fix them quickly. In this episode we interviewed David Cramer about the history of Sentry and how he has built a team around it to provide a hosted offering of the open source project. We covered how the Sentry project got started, how it scales, and how to run a company based on open source.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show, subscribe, join our newsletter, check out the show notes, and get in touch you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry’s real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at getsentry.com and use the code podcastinit at signup to get a $50 credit!- Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we’re interviewing David Cramer about Sentry which is an open source and hosted service for capturing and tracking exceptions in your applications.

Interview with Firstname Lastname

  • Introductions
  • How did you get introduced to Python? – Chris
  • What is Sentry and how did it get started? – Tobias
  • What led you to choose Python for writing Sentry and would you make the same choice again? – Tobias
  • Error reporting needs to be super light weight in order to be useful. What were some implementation challenges you faced around this issue? – Chris
  • Why would a developer want to use a project like Sentry and what makes it stand out from other offerings? – Tobias
  • When would someone want to use a different error tracking service? – Tobias
  • Can you describe the architecture of the Sentry project both in terms of the software design and the infrastructure necessary to run it? – Tobias
  • What made you choose Django versus another Python web framework, and would you choose it today? – Chris
  • What languages and platforms does Sentry support and how does a developer integrate it into their application? – Tobias
  • One of the big discussions in open source these days is around maintainability and a common approach is to have a hosted offering to pay the bills for keeping the project moving forward. How has your experience been with managing the open source community around the project in conjunction with providing a stable and reliable hosted service for it? – Tobias
  • Are there any benefits to using the hosted offering beyond the fact of not having to manage the service on your own? – Tobias
  • Have you faced any performance challenges implementing Sentry’s server side? – Chris
  • What advice can you give to people who are trying to get the most utility out of their usage of Sentry? – Tobias
  • What kinds of challenges have you encountered in the process of adding support for such a wide variety of languages and runtimes? – Tobias
  • Capturing the context of an error can be immensely useful in finding and solving it effectively. Can you describe the facilities in Sentry and Raven that assist developers in providing that information? – Tobias
  • It’s challenging to create an effective method for aggregating incoming issues so that they are sufficiently visible and useful while not hiding or discarding important information. Can you explain how you do that and what the evolution of that system has been like? – Tobias
  • I notice a lot of from future import in Sentry. Does it support Python 3 and/or what’s the plan for getting there? – Chris
  • Looking back to the beginning of the project, what are some of the most interesting and surprising changes that have happened during its lifetime? How does it differ from its original vision? – Tobias

Keep In Touch

Picks

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Buildbot with Pierre Tardy - Episode 57

May 14, 2016

Visit our site to listen to past episodes, support the show, join our community, and sign up for our mailing list.

Summary

As technology professionals, we need to make sure that the software we write is reliably bug free and the best way to do that is with a continuous integration and continuous deployment pipeline. This week we spoke with Pierre Tardy about Buildbot, which is a Python framework for building and maintaining CI/CD workflows to keep our software projects on track.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show, subscribe, join our newsletter, check out the show notes, and get in touch you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • We are also sponsored by Rollbar this week. Rollbar is a service for tracking and aggregating your application errors so that you can find and fix the bugs in your application before your users notice they exist. Use the link rollbar.com/podcastinit to get 90 days and 300,000 errors for free on their bootstrap plan.
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we are interviewing Pierre Tardy about the Buildbot continuous integration system.

Interview with Pierre Tardy

  • Introductions
  • How did you get introduced to Python? – Chris
  • For anyone who isn’t familiar with it can you explain what Buildbot is? – Tobias
  • What was the original inspiration for creating the project? – Tobias
  • How did you get involved in the project? – Tobias
  • Can you describe the internal architecture of Buildbot and outline how a typical workflow would look? – Tobias
  • There are a number of packages out on PyPI for doing subprocess invocation and control, in addition to the functions in the standard library. Which does buildbot use and why? – Chris
  • What makes Buildbot stand out from other CI/CD options that are available today? – Tobias
  • Scaling a large CI/CD system can become a challenge. What are some of the limiting factors in the Buildbot architecture and in what ways have you seen people work to overcome them? – Tobias
  • Are there any design or architecture choices that you would change in the project if you were to start it over? – Tobias
  • If you were starting from scratch on implementing buildbot today, would you still use Python? Why? – Chris
  • What are some of the most difficult challenges that have been faced in the creation and evolution of the project? – Tobias
  • What are some of the most notable uses of Buildbot and how do they uniquely leverage the capabilities of the framework? – Tobias
  • What are some of the biggest challenges that people face when beginning to implement Buildbot in their architecture? – Tobias
  • Does buildbot support the use of docker or public clouds as a part of the build process? – Chris
  • I know that the execution engine for Buildbot is written in Twisted. What benefits does that provide and how has that influenced any efforts for providing Python 3 support? – Tobias
  • Does buildbot support build parallelization at all? For instance splitting one very long test run up into 3 instances each running a section of tests to cut build time? – Chris
  • What are some of the most requested features for the project and are there any that would be unreasonably difficult to implement due to the current design of the project? – Tobias
  • Does buildbot offer a plugin system like Jenkins does, or is there some other approach it uses for custom extensions to the base buildbot functionality? – Chris
  • Managing a reliable build pipeline can be operationally challenging. What are some of the thorniest problems for Buildbot in this regard and what are some of the mechanisms that are built in to simplify the operational characteristics? – Tobias
  • What were some of the challenges around supporting slaves running on platforms with very different environmental characteristics like Microsoft Windows? – Chris
  • What is on the roadmap for Buildbot? – Tobias

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

LibCloud with Anthony Shaw - Episode 55

May 1, 2016

Visit our site to listen to past episodes, support the show, join our community, and sign up for our mailing list.

Summary

More and more of our applications are running in the cloud and there are increasingly more providers to choose from. The LibCloud project is a Python library to help us manage the complexity of our environments from a uniform and pleasant API. In this episode Anthony Shaw joins us to explain how LibCloud works, the community that builds and supports it, and the myriad ways in which it can be used. We also got a peek at some of the plans for the future of the project.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • Subscribe on iTunes, TuneIn or RSS
  • Follow us on Twitter or Google+
  • Give us feedback! Leave a review on iTunes, Tweet to us, send us an email or leave us a message on Google+
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • The Open Data Science Conference in Boston is happening on May 21st and 22nd. If you use the code EP during registration you will save 20% off of the ticket price. If you decide to attend then let us know, we’ll see you there!
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we are interviewing Anthony Shaw about the Apache LibCloud project

Interview with Anthony Shaw

  • Introductions
  • How did you get introduced to Python? – Chris
  • What is LibCloud and how did it get started? – Tobias
  • How much overhead does using libcloud impose versus native SDKs for performance sensitive APIs like block storage? – Chris
  • What are some of the design patterns and abstractions in the library that allow for supporting such a large number of cloud providers with a mostly uniform API? – Tobias
  • Given that there are such differing services provided by the different cloud platforms, do you face any difficulties in exposing those capabilities? – Tobias
  • How does LibCloud compare to similar projects such as the Fog gem in Ruby? – Tobias
  • What inspired the choice of Python as the language for creating the LibCloud project? Would you make the same choice again? – Tobias
  • Which versions of Python are supported and what challenges has that created? – Tobias
  • What is your opinion on the state of PyPI as a package maintainer? What statistics are most useful to you and what else do you wish you could track? – Tobias
  • Could you walk our listeners through the under the cover process details of instantiating a computer instance in say, Azure using libcloud? – Chris
  • Does LibCloud have any native support for parallelization, such as for the purpose of launching a large number of compute instances simultaneously? – Tobias
  • What does it mean to be an Apache project and what benefits does it provide? – Tobias
  • What are some of the most notable projects that leverage LibCloud for interacting with platform and infrastructure service providers? – Tobias
  • Could you describe how libcloud could be extended to abstract away a new type of service that’s not yet supported – e.g. a database? – Chris
  • Would you suggest that libcloud users extend libcloud to cover ‘native’ services they might use like AWS Lambda, or should they mix libcloud and ‘native’ SDKs in cases like this? – Chris
  • Could you talk a little bit about the cloud oriented network services that libcloud supports? Is it possible to create AWS VPCs, subnets, etc using libcloud? – Chris
  • Do you know if people use LibCloud for abstracting the APIs of a single cloud provider, even if they don’t have any intention of using a different platform? – Tobias
  • Do you think that people are more likely to use LibCloud for bridging across muliple public cloud platforms, or is it more commonly used in a hybrid cloud type of environment? – Tobias
  • What is on the roadmap for LibCloud that people should keep an eye out for? – Tobias

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Pip and the Python Package Authority with Donald Stufft - Episode 54

April 23, 2016

Visit our site to listen to past episodes, support the show, join our community, and sign up for our mailing list.

Summary

As Python developers we have all used pip to install the different libraries and projects that we need for our work, but have you ever wondered about who works on pip and how the package archive we all know and love is maintained? In this episode we interviewed Donald Stufft who is the primary maintainer of pip and the Python Package Index about how he got involved with the projects, what kind of work is involved, and what is on the roadmap. Give it a listen and then give him a big thank you for all of his hard work!

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • Subscribe on iTunes, Stitcher, TuneIn or RSS
  • Google Play Music just launched support for podcasts, so now you can check us out there and subscribe to the show.
  • Follow us on Twitter or Google+
  • Give us feedback! Leave a review on iTunes, Tweet to us, send us an email or leave us a message on Google+
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • We also have a new sponsor this week. Rollbar is a service for tracking and aggregating your application errors so that you can fix the bugs in your application before your users notice they exist. Use the link rollbar.com/podcatinit to get 90 days and 300,000 errors for free on their bootstrap plan.
  • The Open Data Science Conference in Boston is happening on May 21st and 22nd. If you use the code EP during registration you will save 20% off of the ticket price. If you decide to attend then let us know, we’ll see you there!
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we are interviewing Donald Stufft about Pip and the Python Packaging Authority

Interview with Donald Stufft

  • Introductions
  • How did you get introduced to Python? – Chris
  • How did you get involved with the Pip project? – Tobias
  • What is the Python Packaging Authority and what does it do? – Tobias
  • How is PyPi / the Python Packaging Authority funded? – Chris
  • What is your opinion on the current state of Python packaging? Are there lessons from other languages and package managers that you think should be adopted by Python? – Tobias
  • What was involved in getting pip into the standard Python distribution? Was there any controversy around this? – Chris
  • Can you describe some of the mechanics of Pip and how it differs from the other packaging systems that Python has used in the past? – Tobias
  • Does pip interact at all with virtualenv, pyenv and the like? – Chris
  • The newest package format for Python is the wheel system. Can you describe what that is and what its benefits are? – Tobias
  • What are the biggest challenges that you have encountered while working on Pip? – Tobias
  • What does the infrastructure for the Python Package Index look like? – Tobias
  • What have been some of the challenges around scaling Pypi’s infrastructure to meet demand? – Chris
  • You’re currently working on a replacement for the PyPI site with the Warehouse project. Can you explain your motivation for that and how it improves on the current system? – Tobias
  • Where do you see the future of dependency management in Python headed? – Chris
  • A few days ago there was a big story about how an NPM library was removed from the index, breaking a large number of dependent projects and applications. Do you think that anything like that could happen in the Python ecosystem? – Tobias
  • What’s on the roadmap for Pip? – Tobias

Keep In Touch

Picks

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

StackStorm with Tomaž Muraus and Patrick Hoolboom - Episode 53

April 16, 2016

Visit our site to listen to past episodes, support the show, join our community, and sign up for our mailing list.

Summary

If you are responsible for managing any amount of servers, then you know that automation is critical for maintaining your sanity. This week we spoke with Tomaž Muraus and Patrick Hoolboom about their work on StackStorm, which is a platform for tracking and reacting to events in your infrastructure. By allowing you to register actions with event triggers it frees you from having to worry about a whole class of concerns so that you can focus on building new capabilities rather than babysitting what you already have.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • Subscribe on iTunes, Stitcher, TuneIn or RSS
  • Follow us on Twitter or Google+
  • Give us feedback! Leave a review on iTunes, Tweet to us, send us an email or leave us a message on Google+
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • I would also like to thank Hired, a job marketplace for developers and designers, for sponsoring this episode of Podcast.__init__. Use the link hired.com/podcastinit to double your signing bonus.
  • ODSC East in Boston is happening on May 21st – 22nd. Use the discount code EP for 20% off when you register
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we are interviewing Tomaž Muraus and Patrick Hoolboom about the StackStorm project, which is an event-driven system automation framework.

Interview

  • Introductions
  • How did you get introduced to Python? – Chris
  • What is StackStorm and what problems does it solve? – Tobias
  • What was your inspiration for creating StackStorm and what were some of the biggest architectural and design challenges? – Tobias
  • What made you choose Python for StackStorm’s implementation rather than another language like Go? – Chris
  • Can you describe the architecture of StackStorm and what the setup looks like? – Tobias
  • Other than chat driven events, what types of event sources does StackStorm support, and what use cases do those alternate event streams enable? – Chris
  • The home page describes StackStorm as being an event-driven framework for automating the users infrastructure. What kinds of capabilities are made possible by this and do you think that it simplifies or complicates the work of operations engineers? – Tobias
  • Is there a minimum or maximum size of infrastructure for which it would make sense to use StackStorm? – Tobias
  • It looks like StackStorm is made up of a number of discrete components. What do the components use to communicate, and how did those choices influence the design of StackStorm’s overall architecture? – Chris
  • I use SaltStack in my work which is a tool that also focuses on event-driven architecture. Can you compare and contrast the capabilities and focus of StackStorm with the features of SaltStack? Would it make sense to use both frameworks in the same infrastructure? – Tobias
  • One of the advertised features of StackStorm is a strong focus on ChatOps. Can you explain that concept for people who might not be familiar with it and describe why it is such a useful paradigm? – Tobias
  • Extensibility is a critical capability for an operations platform due to the wide variety of environments that people are inclined to build. In StackStorm the unit of extensibility is a pack. Can you describe what a pack is and how you arrived at that abstraction? – Tobias
    • Have you encountered any situations in which the concept of a pack has been the wrong abstraction and made something more difficult than it may have been otherwise? – Tobias
  • In very large scale environments like Netflix, how would one build a StackStorm cluster to handle the immense load. More specifically, how does one determine what kinds of machine resources each component needs? – Chris
  • Management of credentials is always a difficult problem in operations. Does StackStorm attempt to tackle that issue or does it defer that responsibility to other systems, such as the user’s configuration management platform? – Tobias
  • Does StackStorm interface with Kibana, Splunk or other log / metric aggregation packages? – Chris
  • What are some of the most surprising uses that you have heard of from people using the platform? – Tobias

Keep In Touch

Picks

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Airflow with Maxime Beauchemin - Episode 44

February 13, 2016

Visit our site to listen to past episodes, support the show, join our community, and sign up for our mailing list.

Summary

Are you struggling with trying to manage a series of related, interdependent batch jobs? Then you should check out Airflow. In this episode we spoke with the project’s creator Maxime Beauchemin about what inspired him to create it, how it works, and why you might want to use it. Airflow is a data pipeline management tool that will simplify how you build, deploy, and monitor your complex data processing tasks so that you can focus on getting the insights you need from your data.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • Subscribe on iTunes, Stitcher, TuneIn or RSS
  • Follow us on Twitter or Google+
  • Give us feedback! Leave a review on iTunes, Tweet to us, send us an email or leave us a message on Google+
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • I would also like to thank Hired, a job marketplace for developers and designers, for sponsoring this episode of Podcast.__init__. Use the link hired.com/podcastinit to double your signing bonus.
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we are interviewing Maxime Beauchemin about his work on the Airflow project.

Interview with Maxime Beauchemin

  • Introductions
  • How did you get introduced to Python? – Chris
  • What is Airflow and what are some of the kinds of problems it can be used to solve? – Chris
  • What are some of the biggest challenges that you have seen when implementing a data pipeline with a workflow engine? – Tobias
  • What are some of the signs that a workflow engine is needed? – Tobias
  • Can you share some of the design and architecture of Airflow and how you arrived at those decisions? – Tobias
  • How does Airflow compare to other workflow management solutions, and why did you choose to write your own? – Chris
  • One of the features of Airflow that is emphasized in the documentation is the ability to dynamically generate pipelines. Can you describe how that works and why it is useful? – Tobias
  • For anyone who wants to get started with using Airflow, what are the infrastructure requirements? – Tobias
  • Airflow, like a number of the other tools in the space, support interoperability with Hadoop and its ecosystem. Can you elaborate on why JVM technologies have become so prevalent in the big data space and how Python fits into that overall problem domain? – Tobias
  • Airflow comes with a web UI for visualizing workflows, as do a few of the other Python workflow engines. Why is that an important feature for this kind of tool and what are some of the tasks and use cases that are supported in the Airflow web portal? – Tobias
  • One problem with data management is tracking the provenance of data as it is manipulated and shuttled between different systems. Does Airflow have any support for maintaining that kind of information and if not do you have recommendations for how practitioners can approach the issue? – Tobias
  • What other kinds of metadata can Airflow track as it executes tasks and what are some of the interesting uses you have seen or created for that information? – Tobias
  • With all the other languages competing for mindshare, what made you choose Python when you built Airflow? – Chris
  • I notice that Airflow supports Kerberos. It’s an incredibly capable security model but that comes at a high price in terms of complexity. What were the challenges and was it worth the additional implementation effort? – Chris
  • When does the data pipeline/workflow management paradigm break down and what other approaches or tools can be used in those cases? – Tobias
  • So, you wrote another tool recently called Panoramix. Can you describe what it is and maybe explain how it fits in the data management domain in relation to Airflow? – Tobias

Keep In Touch

Picks

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Dariusz Suchojad on Zato - Episode 31

November 13, 2015

Visit our site to listen to past episodes, support the show, and sign up for our mailing list.

Summary

Service integration platforms have traditionally been the realm of Java projects. Zato is a project that shows Python is a great choice for systems integration due to its flexibility and wealth of useful libraries. In this episode we had the opportunity to speak with Dariusz Suchojad, the creator of Zato about why he decided to make it and what makes it interesting. Listen to the episode and then take it for a spin.

Brief Introduction

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • Subscribe on iTunes, Stitcher, TuneIn or RSS
  • Follow us on Twitter or Google+
  • Give us feedback! Leave a review on iTunes, Tweet to us, send us an email, leave us a message on Google+, or leave a comment on our show notes
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at
  • I would also like to thank Hired, a job marketplace for developers, for sponsoring this episode of Podcast.__init__. Use the link hired.com/podcastinit to double your signing bonus.
  • Linode is also sponsoring us this week. Check them out at linode.com/podcastinit and get a $10 credit to try out their fast and reliable Linux virtual servers for your next project.
  • We are recording today on October 27th, 2015 and your hosts as usual are Tobias Macey and Chris Patti
  • Today we are interviewing Dariusz Suchojad about Zato

Interview with Dariusz Suchojad

  • Introductions
  • How did you get introduced to Python?
  • Can you explain what Zato is and what motivated you to create it?
  • What makes Zato stand out from other service bus implementations?
  • What are some signs that someone should consider incorporating Zato into their software architecture?
  • Does zato perform well in restricted resource environments like ec2? What performance bottlenecks are common when using zato?
  • It seems that most other ESB projects are written in Java. What advantages does Python have over Java for this kind of project and in what ways is it inferior?
  • The architectural nature of ESBs are such that they form the central backbone of a software system. How have you been able to ensure an appropriate level of reliability and stability in Zato while still delivering new features and improvements?
  • What are the scalability and high availability characteristics of Zato?
  • Does zato run well using pypy?
  • For anyone wanting to use Zato, what are the infrastructure requirements for deployment?
  • What are some of the security ramifications you took into account in zato’s design?
  • What are some of the most novel uses for Zato that you have seen or heard about?

Picks

Keep In Touch

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA