Summary
Dependency management in Python has taken a long and winding path, which has led to the current dominance of Pip. One of the remaining shortcomings is the lack of a robust mechanism for resolving the package and version constraints that are necessary to produce a working system. Thankfully, the Python Software Foundation has funded an effort to upgrade the dependency resolution algorithm and user experience of Pip. In this episode the engineers working on these improvements, Pradyun Gedam, Tzu-Ping Chung, and Paul Moore, discuss the history of Pip, the challenges of dependency management in Python, and the benefits that surrounding projects will gain from a more robust resolution algorithm. This is an exciting development for the Python ecosystem, so listen now and then provide feedback on how the new resolver is working for you.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, node balancers, a 40 Gbit/s public network, fast object storage, and a brand new managed Kubernetes platform, all controlled by a convenient API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they’ve got dedicated CPU and GPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
- You listen to this show because you love Python and want to keep your skills up to date, and machine learning is finding its way into every aspect of software engineering. Springboard has partnered with us to help you take the next step in your career by offering a scholarship to their Machine Learning Engineering career track program. In this online, project-based course every student is paired with a Machine Learning expert who provides unlimited 1:1 mentorship support throughout the program via video conferences. You’ll build up your portfolio of machine learning projects and gain hands-on experience in writing machine learning algorithms, deploying models into production, and managing the lifecycle of a deep learning prototype. Springboard offers a job guarantee, meaning that you don’t have to pay for the program until you get a job in the space. Podcast.__init__ is exclusively offering listeners 20 scholarships of $500 to eligible applicants. It only takes 10 minutes and there’s no obligation. Go to pythonpodcast.com/springboard and apply today! Make sure to use the code AISPRINGBOARD when you enroll.
- Your host as usual is Tobias Macey and today I’m interviewing Tzu-ping Chung, Pradyun Gedam, and Paul Moore about their work to improve the dependency resolution capabilities of Pip and its user experience
Interview
- Introductions
- How did you get introduced to Python?
- Can you start by describing the focus of the work that you are doing?
- What is the scope of the work, and what is the established criteria for when it is considered complete?
- What is your history with working on the Pip source code and what interests you most about this project?
- What are the main sources or manifestations of technical debt that exist in Pip as of today?
- How does it currently handle dependency resolution?
- What are some of the workarounds that developers have had to resort to in the absence of a robust dependency resolver in Pip?
- How is the new dependency resolver implemented?
- How has your initial design evolved or shifted as you have gotten further along in its implementation?
- What are the pieces of information that the resolver will rely on for determining which packages and versions to install? (e.g. will it install setuptools > 45.x in a Python 2 virtualenv?)
- What are the new capabilities in Pip that will be enabled by this upgrade to the dependency resolver?
- What projects or features in the encompassing ecosystem will be unblocked with the introduction of this upgrade?
- What are some of the changes that users will need to make to adopt the updated Pip?
- How do you anticipate the changes in Pip impacting the viability or adoption of Python and its ecosystem within different communities or industries?
- What are some of the additional changes or improvements that you would like to see in Pip or other core elements of the Python landscape?
- What are some of the most interesting, unexpected, or challenging lessons that you have learned while working on these updates to Pip?
Keep In Touch
- Pradyun
- Website
- pradyunsg on GitHub
- @pradyunsg on Twitter
- Paul
- pfmoore on GitHub
- Tzu-Ping
Picks
- Tzu-ping
- Paul
- Pradyun
- because my picks can be anything — things that have kept me sane in this lockdown world
- Music: Chris Daughtry
- Video Game: Parkitect
- because my picks can be anything — things that have kept me sane in this lockdown world
- Tobias
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
- Join the community in the new Zulip chat workspace at pythonpodcast.com/chat
Links
- Pip
- Macdown
- Taiwan
- Pipenv
- PyPI
- TOML
- Python Package Metadata Standards
- iBook G4
- Acorn Computer
- distutils
- easy_install
- Python Eggs
- setuptools
- Python Wheels
- CPAN
- Conda
- Inside The Cheeseshop
- Google Summer of Code
- Zazo
- PEP517
- pip-tools
- Poetry
- resolvelib
- SAT Solver
- Trove Classifiers
- PyPA
- pyproject.toml
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. You're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode. With 200 gigabit in private networking, node balancers, a 40 gigabit public network, fast object storage, and a brand new managed Kubernetes platform, all controlled by a convenient API, you've got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models or running your CI and CD pipelines, they've got dedicated CPU and GPU instances. Go to python podcast.com/linode, that's l I n o d e, today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show.
You listen to this show because you love Python and want to keep your skills up to date, and machine learning is finding its way into every aspect of software engineering. Springboard has partnered with us to help you take the next step in your career by offering a scholarship to their machine learning engineering career track program. In this online project based course, every student is paired with a machine learning expert who provides unlimited 1 to 1 mentorship support throughout the program via video conferences. You'll build up your portfolio of machine learning projects and gain hands on experience in writing machine learning algorithms, deploying models into production, and managing the life cycle of a deep learning prototype.
SpringBoard offers a job guarantee, meaning that you don't have to pay for the program until you get a job in the space. Podcast.init is exclusively offering listeners 20 scholarships of $500 to eligible applicants. It only takes 10 minutes, and there's no obligation. Go to python podcastdot com/springboard and apply today, and make sure to use the code AI springboard when you enroll. Your host as usual is Tobias Macy. And today, I'm interviewing Tsubing Chung, Pradyun Gadam, and Paul Moore about their work to improve the dependency resolution capabilities of pip and its overall user experience. So starting with you, Tsubing, can you introduce yourself?
[00:02:10] Unknown:
So I'm probably most famous of being the author of Macdown, a Markdown editor on Mac. And currently, my occupation is as a freelancer based in Taiwan. And I was also the organizer for Pikang Taiwan for, 2017 and 18. And I maintained Pipem, although I have taken a step back recently because of the work I'm involved
[00:02:40] Unknown:
as a maintainer. And, Pradiyun, how about yourself?
[00:02:43] Unknown:
I'm Pradiyun, as you said. I'm a college student still. I'm the youngest in the group today. I'm a pip maintainer, a moderator on PyPI, a contributor to a whole bunch of open source software, and I guess another thing that's relevant might be Amacodev on Tomo. But, yeah, that's a quick intro, I guess.
[00:03:03] Unknown:
And, Paul, can you introduce yourself as well? Hi. I'm Paul Moore, as has been said. I'm a PIP maintainer. I've been a PIP maintainer for a number of years now. I'm also a CPython core developer, and I am the packaging packaging BDFL delegate for interoperability standards, which is way too many words. But, basically, what that means is that I work with people in putting together standards, like the metadata standards for for Python packages. And I'm in overall charge of sort of running those discussions and basically signing off on the on the final decisions made and the peps that get that come out of it. And going back to you, Zhu Ping, do you remember how you first got introduced to Python?
[00:03:57] Unknown:
Yeah. Well, as, like, as typical of my generation in Taiwan, I started at a fairly, like, convoluted track. So my first programming experience is in college when I got my first ever notebook, which is iBookg4. And at at the time, nobody just like virtually nobody was using Macs. So there are, like Macs are lacking a lot of applications at the time. So I taught myself programming. Like, some of them is in Python, but just basic scripts to automate my workflow. And so I graduated from college and found a job as an iOS programmer because I used to program a lot of Objective C. And so 1 day, my boss just came up to me. Hey. So, the web guy just quits. Do you know anything about Django?
So he's Python. Right? Okay. I can try. And then, like, 10 years later, I'm writing 70, 80% Python, and I haven't wrote a line of virus code since.
[00:05:06] Unknown:
And, Pridi, how about yourself? Do you remember how you got introduced to Python?
[00:05:11] Unknown:
I do. Yes. I think it was about 6 years ago back when I was in school, high school. Dad gave me a book, Copython programming by Wesley Jaichun, and I had too much free time as a kid. And he basically handed me the book, said, hey. Instead of playing that game, make 1. And I was like, oh, okay. Cool. Picked it up, built a game or 2. I was like, hey, this this is fun. And then I started diving deeper into the language itself, sort of seeing CPython exists and and trying to understand what that means and failing at that because I didn't know c by then, obviously. Python was my first language, and that's the only thing I knew. Then I realized I was using this tool called PIP, and, well, then I dove into that, and then
[00:06:01] Unknown:
stuff happened since then. And now I'm a maintainer. And, Paul, do you remember how you first got introduced to Python? I'm not sure my memory is that long.
[00:06:09] Unknown:
I got involved with Python back in the days of about Python 1.4. I was I had an old Acorn computer, back in the day, and it had no software at all on it. So I spent a lot of my time porting stuff to to the Acorn. 1 of the things I looked at was Python, but didn't manage to get anywhere with it. But I came back to it when I when I got my first PC and got involved with the community. And I've I've never used Python as my work. It's always been a hobby for me. So I did a lot of playing around with stuff, got involved with community discussions, got involved with core Python discussions on things like imports.
And that led to, I guess, things like, zip import and then into packaging. Ultimately, to PIP were there were things that I thought I could help with, got involved, and ultimately got invited to be a pit maintainer and never looked back, really.
[00:07:10] Unknown:
So that's how I got where I am today. And for people who weren't around in the earlier days of Python, I know that there were a hodgepodge of different approaches to being able to handle packages and dependencies, and pip is 1 of the packages that came out of that and has become the standard for being able to install things, although there are some competing projects such as conda and easy install. So I'm wondering if you you can maybe give a little bit of the background and history of PIP and how we ended up where we are today with it being the main way that people interact with getting packages onto their system for working with their applications.
[00:07:46] Unknown:
Back in the day, the very original versions of Python, pretty much everything third party was just had a makefile. You you installed it with a makefile, with a c compiler, or you put things together yourself. It was it was very, very difficult. And as a Windows user, it was awful because everything was written for UNIX, and it was way too much work. The first big improvement was when DistUtils came along, which basically standardized the process for building stuff. That may was a huge step forward. It meant that basically people could write their packages and have some level of confidence that anybody could build them. But it was still all very manual in terms of putting things onto your system. You Distutils gave you the bits, but then actually installing them was was still a problem. PIP came along around that time, and, originally, PIP was very much about taking the source code, using Distutils to build it, and installing it. So it was the first real sort of package manager in that sense. But in those days, it was very much, here's a package, get it on your system.
And it had capabilities to do things like uninstall, which was quite a shock to the system in those days because that was that was really user friendly. Around the same sort of time, setup tools was invented, which was a slightly different approach involving trying to deal with managing multiple versions of packages on your system. It was involved a lot of complexity and a lot of sort of cutting edge technology to do clever stuff, but I think I would be right in saying that a lot of what it did was was fairly niche and not many people needed it. So PIP solved a lot of people's problems. It didn't help personally, for me, it didn't help a lot because it was all building from source, and source was still built with Unix compilers and things like that.
But I have no idea when. It a few years afterwards down the line. Daniel Holf invented the wheel package format, which was a format for bringing for building binaries and then just installing them on onto your system. And that was added into to PIP around that time to so that PIP was able to take something that somebody else had built and just put it on your system. And it was at that point, I think, which PIP really started taking off because it made everything so much easier for people who didn't have the capability of of building these things. In parallel with that, PyPI was coming along. There had been various attempts to build package repositories modeled on things like Perl, CPAN, and things like that. But PyPI was the first real centralized place that people could put their packages, and PIP was built to to get packages from there. So the combination of that was a step change for everybody in being able to to get their packages all in 1 go. And that's really where PIP and PyPI came from. Conda that you mentioned was a similar type of effort, but that was built very much by the scientific community to deal with a lot of the specific problems they had around mathematical software, scientific software.
And that developed in parallel and to this day probably remains a sort of parallel ecosystem with a lot of similarities, but it's addressing a slightly different type of problem to to PIP in terms of its its much more specialized. And I guess that's more or less where we are today. There are other things going on. There's been a lot of work being done in recent years to try and standardize things, to try and make it easier for tools to work with each other rather than people having to choose 1 and stick with it. And that's where where a lot of the standards work I'm involved in comes in. And that's that's probably it. Yeah. I got involved, I think, somewhere in the early 2.x
[00:11:44] Unknown:
series of Python, maybe around 2.3. And I can recall having to figure out what Python eggs were and ways to install them and having to figure out how to uninstall them when I got it wrong. And I know that a lot of the ecosystem around packaging for Python and dependency management has just grown up organically, which is where we end up today with the work that you're doing to try and improve the dependency resolution capabilities. And so I'm wondering if you can just start by giving a bit of the background of the work that you're doing and the focus
[00:12:15] Unknown:
of where you're putting your efforts with this current body of work? So the focus of the work that we're doing is fairly well scoped. We're replacing pip's current dependency resolution algorithm with something else that's not broken. That's sort of the core bit that we're doing. The other part of it is we have user experience experts who have been brought on as part of the funding we've raised, and they are working to collect user data, collect user information, analyze it, and sort of work with us as PIP's maintainers and the broader community to improve the CLI, all the error messages, all the reporting in PIP to be more useful for the users.
[00:12:58] Unknown:
And in terms of the overall scope of the work, you said that it's fairly well specified. And I'm wondering what the established criteria are for when it is considered complete and when it's ready to be handed off to the general Python community.
[00:13:13] Unknown:
The criteria would probably be writing the replacement, getting a feature complete, which 1 good metric would be, hey, it passes all of Pip's tests. And then we're gonna have a beta rollout, beta testing phase where we're gonna work with users. And we have user experience folks who can do user testing and have expertise in that area to collect feedback from them, see if the new resolver is actually doing the right thing for the users, is actually better than the existing resolver. The workarounds people have had over the years to work around the broken behavior of the existing resolver, They still work, or they have a clear path to removing those workarounds and things like that. So in terms of criteria, getting a feature complete and rolling it out with a beta phase and then addressing all of that feedback and making it generally available. Making it generally available
[00:14:09] Unknown:
is the this is done point. And I know that there is a lot of technical debt that has been accrued in PIP over the years and that there are some different pain points that people have had from it, dependency resolution being the main 1. And I know that from looking through some of the posts and issues while I was preparing for this interview that there was a fair body of work that needed to be done in advance of getting to the point where you could even start working on addressing the dependency resolution capabilities.
[00:14:39] Unknown:
So I'm wondering if you can talk a bit about some of that advanced work that was necessary and some of the main ways that technical debt manifests in PIP and some of these ways that it has sort of crept into the project over the years? I guess a lot of it originates from the fact that PIP was originally designed to just take everything as source code and then build from there and get to the installed state. And then when wheels were added and sort of static metadata was a concept that was getting added into the code base, that's a transitioning that's probably still happening. In addition to this, as with any software project, the technical debt sort of grew organically.
And at no point did people go, oh, yeah. Let's hold off for a moment and clean stuff up. I mean, that probably happened a bunch of times. There have been major reorganizations, but none where it's, oh, let's remove this functionality and just keep the simple case and things like that never happened. So 1 of the things that we had to do was break out these god classes, these classes that did everything. Right? There was a there is a requirement set class in pip today, which earlier used to do everything, building packages, fetching packages, dependency resolution. All of these happened within that 1 class, and it was, like, a few 1, 000 lines of code. And so 1 of the things that had to be done was breaking this out into multiple pieces that did 1 independent thing and then bringing them together in not the requirement set object, but in the install command so that it's reusable components that we can use elsewhere.
Similarly, there's an install requirement class that does too many things that we're slowly breaking up. But, yeah, it's unifying code flow across the code base so that we're not going through 3 different parts of the code base for the same thing, like, 3 different approaches to building source distributions from directories and think things like that. We're still cleaning a lot of this up, but at this point, we're reasonably good on the rather, at the point where we decided we reasonably well separated the dependency resolution from the rest of the technical debt is when we went, yeah, we should. We can probably, like, expedite this work with some funding.
And, yay, that happened.
[00:17:07] Unknown:
So while while Pradyan was busy doing the restructure and refactoring stuff, I was a maintainer of pipenv. And pipenv was and it's still using pip tools as its dependency resolver. And piptools just uses pip's internal dependency resolver, which is, as we have already discussed, is not very good. So me and 2 other maintainers decide to, hey. Let's just try the brand new 1 for Python because there was no such thing at the time. So we set out to implement a simple backtracking resolver for Pip, and we called it ResolveLib. But, all of us got busy afterwards, and we never really quite finished the project and just set abandoned over there and never got integrated into Pivam.
And 1 day, there's a guy named Pradim just came out to me. Hey. I was doing some Resolver research, and I wrote the thing called Zazzle, and I figured out that it's actually fairly similar to what you were doing. So that's basically how I got onboard to this this body of work. And maybe, Prati, you can take it from there.
[00:18:32] Unknown:
Just wondering what role each of you are playing in this body of work.
[00:18:36] Unknown:
I got involved initially in this space, dependency resolution as a whole in Python packaging, as part of Google's Summit of Code. I basically I had been contributing to Pip for a few months now, and then Google's Summer of Code happened. And I realized I'm eligible because I'm in college now. So I applied. We made the logistics of it work, and I ended up having 3 months of time where I could just work on PIP and, specifically, with the focus on dep improving dependency resolution. That's around the time that I was working on paying down this technical debt. That's when it started. As well as writing Zazzo, as Souping pointed out. So right now, I'm sort of at this point, I've been involved in this getting PIPs, which is dependency resolved, or much better for, like, 2 and a half, 3 years now. So I have a lot of sort of a good understanding of that, very deep understanding of PIPs codebase and how it's quirky and how resolvelib works, how that And I've built these mental models over the years. So I'm sort of contributing that and a bunch of code toward making the resolver happen now. And I guess over to
[00:19:47] Unknown:
Paul. Right. I've I've been involved as a PIP maintainer for quite a while, and I did a lot of work on the p p 517 build implementation for PIP. So I was I was made aware that there was an opportunity to get involved in doing some work on the resolve full time, which basically I I I grabbed at the chance to do that because it it looked like really interesting work to get involved in. I've been picking up the resolver side of things from there, and I'm, I guess, just fitting him, doing chunks of the coding, implementing bits of the features that need to need to be added into the into the resolver implementation.
I've also been trying to sort of keep an eye on the the question of how this fits into the various standards that we've got. Obviously, I've I've got an interest in that. So things like how metadata is provided to PIP, that's something I'm sort of trying to feedback into the wider community, how we how we manage the data that Pip needs in order to do the resolution. So
[00:21:04] Unknown:
I guess that's where I fit in. And also doing a chunk big chunk of the implementation
[00:21:09] Unknown:
along with Soutine. And so in terms of the current state of affairs, what are some of the ways that people might experience the broken dependency resolution of PIP, and how is it currently implemented?
[00:21:21] Unknown:
I take this 1. Go ahead. Basically, for a lot of cases, the current resolver is fine. But what happens is that when a project has multiple dependencies, and particularly when you're trying to install multiple things at once, and the dependencies are either in conflict or are difficult to resolve, PIP's current resolver takes the approach of effectively first come first served. So it goes through in order, says I'm gonna try and make this 1 work. Okay. That worked. Onto the next 1. Onto the next 1. If, as it goes further down the line, it finds something that conflicts with what it's already done, it doesn't backtrack and try again. It just says, oops, and finishes off as best it can. And the result is that for that type of situation, what PIP installs is actually broken. You may have project a depends on version 2 of project b, and what PIP actually installs for you is version 3.
It says, sorry about that, and then you've got to dig yourself out of the mess, which isn't difficult to do because what what you can do is just uninstall the wrong version and install the right 1. But for a user, it's knowing that's happening. It's finding out what needs to be done. It's basically doing the job that you wanted Pip to do. You've got to do manually and pin all
[00:22:48] Unknown:
your dependencies as a result of it. And then as far as the workarounds or resolutions that people can apply manually, what are some of the common practices that people have had to resort to in the absence of a robust dependency resolver?
[00:23:01] Unknown:
A lot of it is that type of thing. It's it's pinning the requirements more tightly than they would like to, particularly in libraries, which is obviously bad. You don't want type requirement pinning in libraries. People are doing things like installing without dependencies and then manually installing the dependencies. There are a number of other projects which have been built up around the packaging ecosystem with the intention of doing the resolution outside of PIP, tools like PIP tools, for instance, does that, Poetry. The project management tool was explicitly built. 1 of its goals was to try and do dependency management on top of what PIP provides.
So there's various different things people do, but at the end of the day,
[00:23:50] Unknown:
it's trying to manually do stuff that PIP ideally should be doing for them. And in terms of the new dependency resolver, how are you approaching that implementation, and what are some of the
[00:24:10] Unknown:
component. We're using a reusable component, resolvelib being that reusable component, which defines an abstraction layer between the dependency resolution algorithm, part that does, oh, let me fetch this 1. Let me see this 1. Let me backtrack this choice and all of that algorithmic logic from the PIP specific details, like here's how I get dependencies of a package. Here's how this package should be represented, and this is what describes a package and things like that. And by separating these details, we're allowing both of them to evolve independently.
So we could slot in a new resolver in the future date that's actually somehow better than the 1 we already have because there are various dependency resolution algorithms in broader space of dependency resolution. So that's 1 of the decision we made is having an abstraction layer so that we can swap the underlying resolver algorithm data if we want to. The other part is we are not trying to make the existing resolver morph into this new resolver. No. No. No. We're gonna implement a new 1 all over again because the structure of code, the the assumptions that the old resolver made that I'm never gonna backtrack are really baked into not the data structures, but the code structure itself. So it just makes a lot more sense to not bother dealing with that bit of debt and just rewriting that, building on top of resolvelib. So we don't actually have to write dependency resolution algorithm, only implement Pip's own details, things like here's how you get metadata and things like that on top of resolve. That's 1 of the design decisions that we made to ignore the existing resolver in the implementation and to make a new 1. The other benefit of this is that we can have a nice rollout approach to this, which is we can have both the resolvers in pip at the same time for users to test with. Right? So that's useful for when we do the rollout because we can just keep improving the new resolver until it's had feature parity slash better than the existing 1 and then flip the switch, once we know it's not gonna be disruptive to do so. K. So, Paul, do you want to mention the
[00:26:27] Unknown:
resolution logic?
[00:26:30] Unknown:
1 thing that people may or may not know is that dependency resolution is is a fairly well known problem. There's lots of things that deal with it. The technical terms are SAT solvers and things like that. So there are libraries that do dependency resolution as a general problem, and many languages already use some form of that. The problem that we have, in particular with Python and with PIP, is that most of those libraries and most of the the sort of algorithms behind the process work on a basis of assuming that they know all the details of the problem upfront.
So I know what packages I'm being asked to install, what dependencies they have, what versions are available. That's all all available to me or at least easy to get. Unfortunately, due to the history of how Python packaging came about, a lot of that just isn't easy to get hold of for Python packages. So, for example, if I'm trying to install a project that's sitting in a directory on my PC, I can't even necessarily know the name of that project without running through a build process. So asking questions like, does this project satisfy this version dependency? Can involve wheeling out your c compiler to an extent.
And that means that algorithms or libraries that work on the basis they can just freely go and check versions are going to be really badly performing for Python and for PIP. So 1 of the big challenges for us was to find a library and an algorithm that would minimize that impact so that we would, as near as possible, only calculate what we needed to know and nothing extra. And that's what that's where ResolveLib really came into the equation because the decision to use ResolveLib was because it basically only got information on demand, and we could use that to feed it with Python data without having that excessive cost.
[00:28:47] Unknown:
Another element that I'm sure factors into the overall complexity of this problem is the existence of self hosted or third party repositories as well where you can't even necessarily rely on the capabilities of what's in the PIPI server of warehouse. And so I'm wondering how that also influences your decisions on how to approach this problem or the capabilities of dependency resolution in those contexts? Probably
[00:29:14] Unknown:
not as much as you might originally think. I mean, the the worst case scenario for for Python and for PIP is installing a project from your local disk, just the source of your the project that's in development. That's a key key use that people have for PIP. And, basically, if we can cover that, then pretty much anything else is easier because the biggest problem that we have in all cases is metadata. It's it's what's the versions, what are the dependencies. And as I say, the good thing about the existing code base for PIP was we already had a lot of that machinery in place. The build process is already there. We just feed it a project and out pops a built version with the metadata we need. So where the problem of having different sources of data does hit us, and it's something that people will have been aware of is when it comes to looking at what projects are valid, what what, for example, wheels, what binaries are valid on a particular environment.
1 case that came up recently was when setup tools dropped Python 2 support. And so in order to install setup tools on a Python 2 environment, you have to pick an older version. Now there's lots of ways of getting that data. Specifically, there's there's 3. There are tags built into the wheel specification, which say this wheel is only usable on Python 3. That's great, but, unfortunately, what that does is it causes PIP to fall back to building from source. So just saying that your wheel is only available for Python 3 doesn't really help because Pip will then say, okay. I'll build from source. Within the standard Python metadata, there is a piece of metadata that says this project works on Python versions x, y, and z. And that is what's used as a sort of final resort because that's the the metadata that we need to build in order to get. So what will happen there is if we build a project and it turns out to be for a version that doesn't match the current environment, the new resolve will at that stage backtrack and try again. So there'll be a bit of a cost in building, but that's it. The old resolve had a big problem here because by the time it had built the project, it had already committed. And so you ended up with people under the old resolver getting an incompatible version installed.
And that is 1 of the things that we wanted the new resolver to fix. The other place where you can get the data that requires Python metadata is also exposed on the package index as part of the data that PyPI provides, and PIP uses that to catch the problem early. That is great. That saves an awful lot of processing from PIP's point of view because it can discard a lot of things straight away. It's the 1 place though where if somebody is using a different piece of index software that doesn't support that data requires Python tag, then they won't get that benefit. And they will have seen with the current version of the resolver, PIP will install things that potentially don't match.
So that's probably the main place where the various different sources is is gonna make a noticeable difference.
[00:32:41] Unknown:
The other sort of important thing here is we are not directly using all the information from PyPI. So there's a I'll need to go in a bit of technical detail, but, basically, there's a page that lists all the versions of a package that are available on PyPI, and that page contains the data requires Python information, like some metadata about the package. Here's the name of the file. Here's what version it is. Here's where you can get the file. Here's a little bit more. But very importantly for dependency resolution, this does not include dependency data. So we do not rely on this page for getting, oh, Flask requires these bunch of packages.
Like, no. No. No. That's not what the resolver sees. What the resolver sees is, okay. These are the available versions of Flask, and that's 1 of them is what I'm gonna get for Flask. And then when we get it locally is when we trigger the build to get dependencies because, essentially, because of history, we have an executable file, a setup.py file where which needs to be executed to get canonical, definitely correct in air quotes, metadata about a package. And that's the build step that Paul's talking about is we need to do the, hey, setup. Py. Tell me about this package step, which is expensive.
And in dependency resolution, when getting dependencies is an expensive step, it's a slow process overall. It takes a lot of optimization to make sure we're not going in the wrong direction, and backtracking will mean that there's gonna be, oh, okay. I made a bad choice. Let me go and do all of that all over again with different set of choices. Right now, we're not gonna enhance any of the interoperability standards for basically letting package indexes like PyPI and other third party package indexes to give this data to pip, partly because that requires writing a PEP and getting a lot of work done because it's a standardization problem, and there's a lot of use cases to consider. People use Python in all sorts of ways. So until that happens, for now, what Pip's gonna do is it's gonna get the package and generate the metadata and get the dependencies. That's gonna be slow. 1 of the big speed ups that we wanna do after the initial rollout of the resolver is working on that as volunteers that we have been for so long, is getting that standardized so that we can make that work for not just PyPI but all 3rd pack package indexes.
And to sort of hint at why we do this is we've been burned by having implementation defined standards in the past where, oh, this works because that's how it's been implemented, and what's implemented is how we do stuff, instead of going, this is how we do stuff, and let's implement that from a piece of text to code. So we've been moving away from that through the standardization efforts that the Python Packaging Authority and the broader Python community has been doing in this space. And yeah. So we are not gonna introduce more implementation specific stuff, implementation defined stuff, while also working towards reducing doing that. So that's why we're not gonna use information that PyPI does provide in, like, a PyPI specific API until we standardize it and make it possible for third party package indexes to expose the same information.
[00:36:20] Unknown:
Yeah. And going back to Paul's example of setup tools, it's something that I've personally been burned by because I've got a Python 2 project that I have to maintain at my work and, you know, creating the virtual end for Python 2 and then having it install setup tools greater than 45 and then just having everything break and having to manually override that. So I'm definitely looking forward to this new resolver being mainlined into PIP. So keep up the good work on that front. And then so in terms of the new capabilities that you're hinting at and some of the new standards that you're looking forward to, what are some of the overall improvements in PIP itself and some of the surrounding packaging ecosystem that you either anticipate or that are explicitly blocked by this work? So 1 of the big ones is
[00:37:07] Unknown:
better environment management in PIP, which has been something that lots of folks have been asking for. The other is the dependency re resolution logic is now gonna be in a shared library. And so not everybody in this space will be implementing their own, and we'll have at least some way to do interoperability. Although this is implementation defined, but it's also a lot closer to the implementation. So the other ways it'll help simplify some of the other tooling in the ecosystem. So as Sipping hinted, pipenv depends on pip tools, which depends on pip's internals. If pip's internal resolver improves, pip tools improve, and pipenv improves.
Similarly, other projects that are using pip under them for whatever reason, they benefit from this work as well. And another thing that's been very common is being able to upgrade packages without breaking the existing packages, which I want a newer version of Django, and, oh, no. Now my extensions don't work. Now my plug ins don't work. Now my other applications don't work. Like, that should not happen. And that's 1 of the things that this was all will enable in some senses, although that's something it should be doing already, but it's not.
[00:38:23] Unknown:
So. That's definitely something that I've been burned by as well. And I'm sure many people have also have running pipdashu for a particular package and having it bring along a half dozen other things that you didn't anticipate?
[00:38:34] Unknown:
Yeah. That got fixed at some point partially, but it was more of a workaround. It was it stopped upgrading everything it saw and only things it needed to. But, again, that wasn't ideal, and doing a proper result solves that much better and more correctly.
[00:38:50] Unknown:
And, is somebody who is maintaining 1 of the projects that is dependent on PIP. What are some of the improvements in pipenv or some of the other projects that you maintain that you are excited to be able to get started with once the resolver ends up in mainline pip? So
[00:39:06] Unknown:
as we previously mentioned, pipm currently depends on pip tools, which depends on pip. But 1 of the, I guess, optimizations pip is doing in this dependence resolution logic, it's it only resolves the dependencies for the current environment. So for example, if you depend on, I don't know, Django 3.0 on Linux, but you, for some reason, want to depend on Django 3.1 only on Windows, then the current PIP resolver, even the new PIP resolver, will only choose either a 3.0 or 3.1 based on what platform they're currently on. But for pipenv, for for for example, Poetry or maybe PIP tools, the ideal scenario would be to generate an I call the abstract dependency tree. So you will need to generate a log file that says, hey, if Windows, then install Django 3.1. Otherwise, Django 3.0.
And this is 1 of the things I am looking forward to do by, like, learning with while I'm learning from the work I'm doing in pip, I can translate all the knowledge to building a better abstract dependence resolver for prepayment and other similar tools.
[00:40:27] Unknown:
And as part of the overall project, you mentioned too that you have a user experience team who are conducting user interviews. And I'm wondering how some of that has fed back into your plans for this project and some of the ways that it has influenced your original ideas or changed the direction that you're taking for the the technical implementations.
[00:40:48] Unknown:
Yeah. So if I start from resolvelib so because resolvelib was kind of developed in the back in the vacuum without a actual use case until PIP decided to choose it. 1 of the things that Resolve. It does not do well is error reporting because we don't have real world example to report on. So 1 of the biggest advantage the the user research team are feeding back to us is how people are thinking about PIPs error reporting on dependencies and how we can do it better potentially in the new resolver.
[00:41:27] Unknown:
I think 1 of the other things that has been very valuable, certainly speaking as a PIP maintainer, is just simply getting a view on how people are using PIP out there. I mean, obviously, we we see reports of what pea what people are doing, but a lot of what we get in the normal course of events comes from people raising issues. So, optimistically, I assume that we're getting a fairly biased view of how bad Pip is from from only ever seeing bug reports. And 1 of the things we've we've definitely got from talking to to some end users about how they use PIP, what they use it for, how it works, I was actually quite surprised that a lot of people were finding that they wanted PIP to be more strict, more definite, more precise in what it it did, and they weren't trying to do bizarre, weird, and wonderful things, and they wanted to force Pip out of its comfort zone.
They were actually just trying to get on with fairly straightforward stuff, and they just wanted PIP to continue doing what it did but better. So that that was a really a really sort of positive bit of feedback and also reassuring that when you're trying to deal with a a problem like dependency resolution, you spend your life mired in ridiculously complicated examples of 20 things all all independent all interdependent on each other and all conflicting, and how are we gonna deal with this problem. And getting that step back and getting the feedback that most people have relatively straightforward environments with relatively straightforward problems they wanna see fixed. And if we do that, we've solved a lot of the problems.
That puts a a great perspective on on what we're trying to achieve and make makes it makes it more achievable, ultimately.
[00:43:21] Unknown:
Yeah. And, yeah, I I just strongly agree with what Paul said that with the Pip Maintainer hat on, like, I think the most valuable bit of information has been some amount of improved visibility into how users are using PIP. Because to put it mildly, a lot of people are using PIP, and we generally don't see a good chunk of, hey. PIP just worked because the normal communication channels are biased towards, hey. PIP did not work, or, hey. I want PIP to do this other thing as well. So getting that sort of perspective, as Paul put it, has been super useful. The other bit has been some amount of better not sure what the word would be. Better mental model for approaching user facing changes as part of learning about these topics from people who are experts in the space and sort of getting, at least for me personally, getting their take on how to handle some of these disruptive changes that we might make in the future or sort of communication around those and how to handle
[00:44:28] Unknown:
telling users about these and so on. And that has been useful as well. So hopefully that translates into slightly nicer experiences for users moving forward. And when it comes time for people to start using the new resolver, is there any work that they're going to have to do on their end or changes to their workflow? Or is it something that will just land in mainline and it should essentially be invisible to end users other than the fact that they'll start getting fewer errors from their installs?
[00:44:56] Unknown:
It should be as simple as that. What we will the the plan for the rollout is that right now, the new resolver is available using a flag within PIP. So you can enable the new resolver by saying dash dash unable unstable feature equals resolver, and you can see how it's working and how it's progressing. It's obviously only very much in an alpha state now. At the point where it's finally released, that unstable feature flag will no longer be needed, and the new resolve will just be there, hopefully. And the the goal is that that will have as little impact as possible. But, obviously, as we talked about, people will have been working around existing issues.
People may have breakages in their environment that PIP has installed things with conflicts. And so we're going through a process at the moment, and we'll be continue there's gonna be a a proper beta release of the new resolver, which will be publicized later in May, I believe. And at that point, we're looking for people to actually try and exercise a new resolver if possible or more generally run PIP check on their environment to make sure that they don't have issues in their existing installation, which when the new resolver comes along, it might say, I'm sorry. I don't see how you could possibly have this. I'm not gonna deal with it. Because, obviously, the new resolver is designed to avoid conflicts. And if you hand it an already conflicting environment, it's gonna have problems.
So a little bit of prework on the part of users to make sure their environments don't have such problems in terms of maintaining their projects, looking at how they're specifying on their dependencies, are they doing anything at the moment to work around resolver issues, and what's their plan going to be for maybe getting rid of that? The the workaround should carry on working, but they'll rather than being necessary, they'll change to being suboptimal. So, ideally, people should look at planning to phase them out once the new resolver is in place. So there are things people can do, and it will it will help both them and us to to get the new resolver in place. But for, hopefully, the vast majority of people, just enjoy the benefits of not breaking your dependencies for you.
[00:47:31] Unknown:
And then in terms of the broader ecosystem and broader experience of people using pip and programming in Python, how do you anticipate these improvements in pip impacting the overall viability of the Python ecosystem and its use within different communities or industries that might be shying away from it because of some challenges that they faced in dependency conflicts?
[00:47:57] Unknown:
Hopefully, I think the main thing would be that I think if you take it from the other angle, at the moment, dependency resolution in Python is known to be not perfect, should we say. So there will be people that are looking at their project and saying, should I use Python? We've got an awful lot of dependencies. Is it going to be okay? And, hopefully, having a better story around how the resolve works and dependency resolution mean that people will be more confident in saying, yeah. I can choose Python because dependencies aren't a problem. So, I mean, I I don't I don't want to sort of make it sound like we're going to solve all the problems of the universe here, but a little bit more enthusiasm for Python because it doesn't have as many concerns around it, will hopefully improve adoption.
It would be nice to think that the improvement in supporting tools that we were talking about previously, that will just generally make for a nicer experience in in the overall ecosystem, but that's obviously a little bit longer term. And I guess on a a sort of very personal level, speaking with my standards guy hat on, seeing how PIP can use good dependency data to produce a good result will encourage the community to think more in terms of providing that data, in terms of publishing the metadata statically, not building it on the fly, not sort of trying to say, well, we need this here, we need that there. If if we can put all of that in a form that Pip can just grab in 1 go, that that will be yet further improvements. Having that data available will be will be really useful. And I think the the new resolver, in my mind, will be a good showcase of how more data, more metadata, better metadata can improve the tools that people are using, and so will bring people's thinking forward on that score.
[00:50:05] Unknown:
Yeah. And the other thing that in addition to the standardization, the other thing is the tertiary effect, which Paul is sort of deemphasizing of, like, getting more consistent information and good information, good metadata from existing projects over time. I think in the longer term, that'll be the bigger, more relevant effect of it since it'll push projects to be more correct about, hey, this is what I depend on, and that'll help reduce problems with the metadata itself, which right now PIP masks completely. And it will start surfacing those issues with the new resolver, and those will start, in the longer term, getting fixed. In the short term, they'll be, like, a bit of a pain point during the rollout, possibly. I am not sure if that's what the rollout is for, figuring out.
But, yeah, in the longer term, in general, it'll push for nicer metadata, more static metadata, all of which will be bringing Pip more towards
[00:51:12] Unknown:
the nicer packaging experience than it is. It's been a it's been a long road, and this is another step on that road. And I think looking to the future, we will be continuing down that process for some time yet. But it's it's, it's definitely another case of us going in the right direction in my view.
[00:51:33] Unknown:
In terms of your own goals, once this body of work is done, what are some of the additional changes or improvements that you would like to see or be involved with either in pip or some of the other core elements of the Python landscape?
[00:51:47] Unknown:
So I think we've already mentioned a couple. 1 of them being static metadata. Basically not having to do set it up by give me information step and just going, okay. I can read this file and get information. So putting that metadata in Pyproject. Toml like, Poetry is already doing that today, but we wanna make it possible for it to be done by all the tools. Right? So having an interoperability standard about this. And there is some discussion happening. Poetry authors are involved. PIP authors are involved. Certitude authors are involved. We're all discussing this right now, and I expect that'll happen in the near future, although no promises on the timeline because it's all done on a volunteer basis. The other is that we have already mentioned also is being able to standardize away for pip and other Python packaging installers and environment managers or whatnot to get dependency information from a package index, whether it's PyPI or someone's Artifactory instance or something else entirely.
Being able to get that information into the installer without needing to download, do a build, look at the results. And the third 1, which is a lot more, broader, is move further in the direction of having reusable libraries for doing these chunks of jobs in Python packaging, so going from a wheel to installing the wheel on the system, well, you don't really need pip to do that. It's a well defined step. Maybe we could have a library that pip uses that actually does this, and then other people can use that library. Similarly, building packages, it's a well defined process.
It's just that pip contains the only good implementation of it. Maybe if we make a common implementation that pip also uses, everybody can use just that step. And sort of moving to these reusable libraries to decouple the ecosystem further from the implementation defined details of pip, of setup tools, and so on, to being more general purpose and being more of wrappers around these libraries that handle all of the legacy options that they have and so on and so forth and then letting newer tooling evolve, potentially replacements, potentially improvements to PIP set of tools, whatnot, themselves that essentially allow for the evolution of the ecosystem as a whole because that's the goal, right, to make things better as we go. And these reusable libraries and
[00:54:34] Unknown:
will help with that as well. Yeah. I want to second, yeah, I want to second Pradyan's last point of reusable libraries. A lot of Pip's deficiencies came from the fact that PIP is a 10 year old project that was designed to do something else than what it is currently doing. So to so a lot of the new packaging projects in Python are able to solve this problem better simply because they don't have the same history baggage pip has. But since but right now, PIP is the only implementation that can do everything PIP can do. So by splitting out the the u reusable parts, it will be much easier for the community to come up with different solutions that fits the modern world better than Pip, while Pip can keep on doing what it does best and support those people that are already using Pip for niche applications.
[00:55:45] Unknown:
Paul, do you wanna add something? I could I could thank you guys for being my standard cheerleaders. I think everything everything you said is is essentially correct. With my role as standards person, that is absolutely my goal is to make sure that the ecosystem is built in such a way that if somebody wants to come along and write a specialized tool for their particular use,
[00:56:15] Unknown:
they don't have to reinvent all the wheels that Pip has invented over time. I mean, we already reinvented a wheel format, so we don't wanna do more of that.
[00:56:26] Unknown:
Pradyan. Pradyan. Though I promised my family I wasn't going to make bad puns on this podcast. You've just ruined it for me.
[00:56:34] Unknown:
I'm sorry.
[00:56:35] Unknown:
But, yeah, I I I I I'm gonna collapse in disarray now. Essentially, yes. Let's standardize stuff. Let's make it reusable. Let's give the community a chance to innovate in ways that it can't when everything's getting measured against. I'd love to use your tool, but it doesn't do this particular bit of what PIP does. And then in terms of your experiences
[00:57:03] Unknown:
of working on this new dependency resolution algorithm and improving some of the overall implementation of pip and paying down its technical debt, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
[00:57:17] Unknown:
For me, I think I mentioned earlier, this is the first time I've ever actually been paid to work on Python. My my my day job does not involve Python except in very minor ways. And for me, oddly enough, the thing that I found remarkable was how much how different it was working on PIP as a full time well, in my case, part time, but nevertheless, dedicated piece of work rather than doing it as a hobby in my spare time. The ability to to focus on the bigger problems, the ability to get deeply involved in addressing the difficult issues rather than finding an hour or 2, 1 evening to to to knock off a couple of little bits that have been bugging me. That's been that's been remarkable how much how much of a difference it's made. And I I don't think I had realized even as even while I have said that 1 of the problems PIP has is a lack of resources. We've got I'm not quite sure exactly how many maintainers we've got now, 5 or 6, all of whom are volunteer working on it in their spare time. I don't think I'd realized how much of an impact that has on what what we've been able to do with PIP. If you look at a lot of the other language communities and their packaging solutions, a lot of them have funded work going on for them. If you look at Conda within the Python environment, that's got I believe got corporate support, but what Pip's doing is is very much spare time. So so actually being the the the really unexpected side for me was just realizing how much more productive it could be if we could get people working as a dedicated task on Python packaging.
[00:59:09] Unknown:
Yeah. 1 of the more interesting things for me personally and is, like Paul, this is the first time I'm working on a project, getting paid to work on a Python project. It just so happens that it's also the first time I'm probably getting paid to do work because I'm fresh out of college, actually still in college. And I think over the past few years, 1 of the things that's been super interesting for me as a contributor is just seeing how different the ecosystem funding situation is. Because if I look at other package managers, npm, cargo, bundler, all of those have some sort of big financial backing organization that's paying people full time to work on these things. And people coming from other toolings, other ecosystems come in expecting pip is working the same way.
And for a little bit there, the reality of it was it was just a college student sitting in his dorm room and fixing those issues or responding to the issue comments. And it's it took me a while after actually having started working on this to go, wow. That was the situation this tooling was in, and that's not great. But the other part of it is it's been awesome to actually have funding to do this stuff. Right? This is the first time that PIPs ever had funding go towards it, and kudos to everybody in the Python Software Foundation's packaging working group who made this happen. Yeah. It's just really different approach to working. The feedback loops are faster, at least compared to what they were earlier. They're not instant or super fast, but it's really nice to make it faster. The other probably thing I have learned would be just the sheer number of ways people use Pip. In the 3 years that I've been contributing slash maintaining it, I have got a glimpse of it. But to actually have user surveys and see the results of those and sort of get that information in data form rather than a qualitative form has been really enlightening, to put it in 1 way, I guess. And the face to face time with other contributors has been really nice because, as I said, there's not been much resources.
So we have not actually, like, spent time discussing over calls about issues. Most of it has just been collaboration over comments on GitHub and mailing lists and discuss dot Python dot org. And to have, like, the ability to go, hey. Do you have a minute to hop on a call and discuss this real quick? And getting a response within the an hour has been basically amazing compared to needing to wait for 3 days to get a response to a detailed comment you made because it was too long to read for someone in the 10 minutes they had that day. It's yeah. That's been a big change and really nice.
[01:02:12] Unknown:
So for background, I'm the only 1 of the 3 that was not a PIP maintainer before joining the project. So my so my impression to the project is entirely opposite in a way. So this is not my first freelancing work. So I've been doing remote work for a while, but this is the first time I'm working with people real time in different time zones, like literally half the globe away. Almost everyone in the team is in a different time zone. Yeah. I believe the 4 of us are in different time zone right now. So both Paul and Prati mentioned the feedback loop was fast. For me, the feedback loop initially was terribly slow because everyone's in different time zones, and they are all working part time. Or because we are dealing with different parts of PIP, and even though we have 2 PIP containers, they don't necessarily have knowledge to all of the codes in PIP.
So sometimes we need feedback from other PIP maintainers. And, like, comparing to a corporate setting, like, you can just walk by or just tell me, hey. Maybe I would drop by tomorrow morning and see what we can work out when this doesn't happen with this project. So this has been a renovation to me. And it is also very, I think, surprising is a wrong word. I'm very impressed how everyone has managed this project with so much restrictions going on, and it really takes everyone a lot of discipline, independence, and self consciousness to push things forward.
[01:03:54] Unknown:
Alright. Are there any other aspects of the dependency resolution work that you're doing or your experiences working on pip or your experiences of contributing to the overall Python community that we didn't discuss that you'd like to cover before we close out the show? I think 1 of the nicer effects
[01:04:10] Unknown:
of this project actually happening along with PyPI rollout happening earlier is sort of acting as a showcase to the Python packaging ecosystem that, hey. We can actually get funding for open source projects to do stuff with. Like, if we can show people that, hey. This money will be put to good use. It's possible for us to get and raise funding for targeted open source projects, which is a really good model for pushing open source tooling forward. Because for end users, there's a clear expectation in what the funding should result in as an output for them, which is, let's be honest, how most funding works, and no one's gonna give you money to do just whatever you want. And on the other hand, that funding can be used to improve maintainer availability, developer time availability, and improve the project not just in that 1 dimension but also other dimensions. Right? Put that money towards general maintenance as well. We've had a release process while I've been getting paid to work on PIP.
And this release went fairly smoothly compared to the past ones because I had a lot more time to pay attention to details. And things like those, are reasons that we should get funding into open source. And I think the model that this project shows of, hey. Have a targeted project, raise funding for it, and get the existing maintainers to work on it, is perhaps not sustainable, but definitely better than not having funding at all. I think 1 other thing I would say,
[01:06:03] Unknown:
this is everybody says this. I'm going to be so unoriginal. It's not true. But it's been really, really enjoyable working on this project. I think the people in the pipe and community, the people we've dealt people I've dealt with make it so much fun. I've had a blast, and that is all down to the community as a whole. The support we get from the people using PIP, the support we get from the people working with us, it's it's amazing. And I think that's something that I think it's important to remember. We've got an amazing community, and it makes such a difference.
[01:06:44] Unknown:
Yes. Definitely. Yeah. Definitely.
[01:06:47] Unknown:
It's it's been so nice to get, like, random tweets from people like, hey. This beta like, the alpha resolver works well. Yay. And it's like, it makes your day. It makes you really happy that stuff that you're doing has almost immediate impact on users, and they're super stoked about and about it and also motivating you and telling you, hey. Good stuff. It's really nice. I interrupted something. Sorry. No. I was just saying
[01:07:13] Unknown:
I very much agree what Paul said. I I mean, I'm not going to say that it's been a blast. Let's do it again, but maybe we can do it again for something else.
[01:07:25] Unknown:
How much of that is because we've had to deal with a lot of technical debt, though?
[01:07:30] Unknown:
I don't know. Maybe it's just very nice to have someone to talk to when you're hitting with, like, the technical devs. Like, you're not alone in this. There are other people also suffering from the same problems as you are. Yeah. So it's a very good experience.
[01:07:48] Unknown:
Alright. Well, for anybody who wants to follow along with the work that you're all doing or get in touch or get involved, I'll have you add your preferred contact information to the show notes. And with that, I'll move us into the picks. And so how about we start with you, Tzu Ping? The first 1
[01:08:04] Unknown:
is py launcher from Brett Kennon, a core developer and also a a member of steering wheel console. Py Python launcher or Py launcher. I forgot the name. I'll provide a link later. So it's a project in Rust that when you type py in your commands, it dynamically loads up all the pythons in your system and launch the best 1 for you, and you can supply flags to it to let it choose maybe Python 2 and Python 3.5 specifically. So if you're familiar with Windows there, Python has had a similar thing, Copypy, the launcher for Windows py.exe, which loops in the registry for install pythons and launch 1 for you based on the arguments you passed it in. And, and but we don't really have something similar on Macs or Windows. So sorry. For for Linux.
And that has been the problem in tutorials because you need to provide different commands in like, for different platforms. And Python launcher is a project that aims to solve this problem so we can have 1 canonical way to launch Python. And the other 1 is is a book author, Joe Abercrombie. He writes fantasy novels. I've been reading his Shattersea trilogy, and it was it's been very interesting. And I've recently picked up the other series he had. I forgot the name. I forgot the name. And it's I'm I'm also really looking forward to it. And if you are into these kind of novels, fantasy novels, I highly recommend this writing. And the final 1 is maybe anime in general.
So I know anime is not a big thing outside of Eastern Asia, but and a lot of people think that anime is for kids, like cartoons. But comparing animes to movies is kind of like comparing Python to Windows, which does not really make sense. 1 of 1 of them is an art form and 1 of them is a media. So if you're like, if you have been staying home for too long and are looking for something to pass time, I recommend, like, maybe watching maybe watching some more adult themed animes, and maybe you will find something suitable for you. And, Paul, do you have any pics this week? I guess 1 of 1 of the things that I've found
[01:10:41] Unknown:
has been really useful for me, I've as part of this project, I had set up a brand new laptop And a couple of really nice tools that I've enjoyed using for that are on the Python side. There's a project called pipx, which basically lets you set up Python projects that work as sort of command line tools, things like black, things like Knox, tox. You can set them up as local little commands. They've got their own isolated environment. You don't have them installed within your own main environment, and they just run like independent commands. It's really great for for managing all of those tools that make a PC turn into a usable working environment.
And a similar thing that I use, I'm based on Windows and finding all the little tools, things like Unix utilities, command line tools that that you want to use. There's a really nice package manager that I've discovered and use a lot, which is called Scoop, which effectively manages you installing and uninstalling all those little tools that you you wanna get hold of just from a simple command line interface. It's really helped me set things up fast and and not have to worry about why am I on a PC that hasn't got such a thing installed again. So though those are really great. I've thoroughly enjoyed them.
And I guess following on from the theme of sort of what do you do while you're in lockdown, A guy I've read a lot of books by is Neil Gaiman, who I guess a lot of people will have heard of. His books, I think, are really amazing. He's done a lot of work on TV as well. I've recently watched the TV series of Good Omens, which is a thing he did with Terry Pratchett, which I thought was absolutely fantastic. I thoroughly enjoyed watching that. I'd recommend it to anybody who enjoys our either Terry Pratchett or or Neil Gaiman. Well worth a look. Since Paul just mentioned Scoop and
[01:13:03] Unknown:
PaypalX, I made a, Scoop formula. I I forgot the name recipe Scoop recipe for PaypalX, so you don't need to globally install Pipex to your Python. Excellent.
[01:13:19] Unknown:
Well, Paul picked up the PIPEX, which was gonna be 1 of my picks. So I guess I'll keep going with the things that have kept me sane in this lockdown world that Paul picked up. I think 2 of them have been re mostly recreational things because I've had enough work stuff going on to keep me occupied, luckily. So it's been music by Chris Daughtry, who has been around since quite a while. He was, like, an American Idol, contestant in 2006 and so on, and he's, like, an alternative rock band. Chris Daughtry's band is an alternative rock band, and it's a nice bunch of music. They have a really good discography. I don't know of a single song in there that I don't like, So that has kept me sane.
The other thing is probably being Parkitect, which is an indie game where you build and manage an amusement park, basically. And it's sort of a really good time way to put down a whole bunch of creative energy in 1 place, and build stuff that at least makes me feel happy about them. So that's another.
[01:14:33] Unknown:
Alright. And for my picks this week, I just recently started experimenting with the language server protocol and its implementation in emacs with LSP mode. And it's just a way of being able to have 1 way of sharing a lot of the IDE like behavior between different editors. So things like, completions and linting and, being able to navigate definitions or find references to different variables. So, definitely a pretty great way of being able to have your environment easier to manage and, portable across different editors. So been enjoying that and, recommend that for anybody who's using any sort of text editing environment. And so with that, I will close out the show. I just want to thank you all for taking the time today to join me and share the work that you're doing on improving the dependency resolution and pip. It's something that is going to benefit everybody in the community and something that I look forward to taking advantage of myself. So I appreciate all of your time and effort on that, and I hope you to enjoy the rest of your day.
[01:15:40] Unknown:
Thank you. Thank you. Thanks, Tobias. Thanks, Tobias. Have a good day, folks. Right. Bye bye. Bye, everyone.
[01:15:49] Unknown:
Thank you for listening. Don't forget to check out our other show, the data engineering podcast at data engineering pod cast.com for the latest on modern data management. And visit the site at python podcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host@podcastinit.com with your story. To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.
Introduction and Sponsor Messages
Meet the Guests: Tsubing Chung, Pradyun Gadam, and Paul Moore
First Encounters with Python
History and Evolution of Pip
Current Work on Dependency Resolution
Technical Debt and Refactoring in Pip
Challenges with Current Dependency Resolution
Design and Implementation of the New Resolver
User Experience and Feedback
Impact on the Python Ecosystem
Future Goals and Improvements
Lessons Learned and Challenges Faced
Funding and Community Support
Picks and Recommendations