A majority of projects will eventually need some way of managing periodic or long-running tasks outside of the context of the main application. This is where a distributed task queue becomes useful. For many in the Python community the standard option is Celery, though there are other projects to choose from. This week Bogdan Popa explains why he was dissatisfied with the current landscape of task queues and the features that he decided to focus on while building Dramatiq, a new, opinionated distributed task queue for Python 3. He also describes how it is designed, how you can start using it, and what he has planned for the future.
Do you want to try out some of the tools and applications that you heard about on Podcast.__init__? Do you have a side project that you want to share with the world? With Linode’s managed Kubernetes platform it’s now even easier to get started with the latest in cloud technologies. With the combined power of the leading container orchestrator and the speed and reliability of Linode’s object storage, node balancers, block storage, and dedicated CPU or GPU instances, you’ve got everything you need to scale up. Go to pythonpodcast.com/linode today and get a $100 credit to launch a new cluster, run a server, upload some data, or… And don’t forget to thank them for being a long time supporter of Podcast.__init__!
With GoCD’s comprehensive pipeline modeling, you can model complex workflows for multiple teams with ease. And GoCD’s Value Stream Map lets you track a change from commit to deploy at a glance.
GoCD’s real power is in the visibility it provides over your end-to-end workflow. So you get complete control of and visibility into your deployments, across multiple teams.
Say goodbye to deployment panic and hello to consistent, predictable deliveries.
To learn more about GoCD, visit gocd.org for a free download. Professional Support and enterprise add-ons, including disaster recovery, are available.
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable.
- When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at podastinit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your awesome app. And now you can deliver your work to your users even faster with the newly upgraded 200 GBit network in all of their datacenters.
- If you’re tired of cobbling together your deployment pipeline then it’s time to try out GoCD, the open source continuous delivery platform built by the people at ThoughtWorks who wrote the book about it. With GoCD you get complete visibility into the life-cycle of your software from one location. To download it now go to podcatinit.com/gocd. Professional support and enterprise plugins are available for added piece of mind.
- Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email email@example.com)
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Your host as usual is Tobias Macey and today I’m interviewing Bogdan Popa about Dramatiq, a distributed task processing library for Python with a focus on simplicity, reliability and performance
- How did you get introduced to Python?
- What is Dramatiq and what was your motivation for creating it?
- How does Dramatiq compare to other task queues in Python such as Celery or RQ?
- How is Dramatiq implemented and how has the internal architecture evolved?
- What have been some of the most difficult aspects of building Dramatiq?
- What are some of the features that you are most proud of?
- For someone who is interested in integrating Dramatiq into an application, can you describe the steps involved and the API?
- Do you provide any form of migration path or compatibility layer for people who are currently using Celery or RQ?
- Can you describe the licensing structure for the project and your reasoning?
- How did you determine the price point for commercial licenses?
- Have you been successful in selling licenses for commercial use?
- What are some of the features that you have planned for future releases?
Keep In Touch
- Google App Engine
- GAE Task Queue
- LRU (Least Recently Used)
- SQS (Amazon Simple Queue Service)
- Google Cloud PubSub