Debugging Python Projects With PySnooper - Episode 241

Summary

Debugging is a painful but necessary practice in software development. The tools that are available in Python range from the built-in debugger, to tools integrated with your coding environment, to the trusty print function. In this episode Ram Rachum describes his work on PySnooper and how it can be used to speed up your problem solving in complex or legacy applications.

Do you want to try out some of the tools and applications that you heard about on Podcast.__init__? Do you have a side project that you want to share with the world? With Linode’s managed Kubernetes platform it’s now even easier to get started with the latest in cloud technologies. With the combined power of the leading container orchestrator and the speed and reliability of Linode’s object storage, node balancers, block storage, and dedicated CPU or GPU instances, you’ve got everything you need to scale up. Go to pythonpodcast.com/linode today and get a $100 credit to launch a new cluster, run a server, upload some data, or… And don’t forget to thank them for being a long time supporter of Podcast.__init__!



Announcements

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, or running your build servers, they just launched dedicated CPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media and the Python Software Foundation. Upcoming events include the Software Architecture Conference in NYC and PyCon US in Pittsburgh. Go to pythonpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
  • Your host as usual is Tobias Macey and today I’m interviewing Ram Rachum about PySnooper, an alternative approach to debugging your python projects

Interview

  • Introductions
  • How did you get introduced to Python?
  • How do developers normally debug their code, and what need does PySnooper address that isn’t addressed by the established methods?
  • What is the workflow for using PySnooper for investigating or debugging a project? (This will probably be answered in the answer to the question above)
  • What are some of the pieces of information that it surfaces and how do they aid the developer in directing their investigation?
  • What were some of the projects that you were testing it with and how did they influence the direction that you took PySnooper?
  • Can you describe how PySnooper is implemented and some of the ways that it has evolved since you first began working on it?
  • What are some of the initial goals that you had for the project which you have since abandoned as either not useful or too challenging to implement?
  • What are some of the edge cases or technical challenges that you have encountered while working on PySnooper, either in Python itself or in the tool?
  • There is another project called Snoop which builds on top of your work on PySnooper to add some extra functionality and developer ergonomics. What, if anything, was your reaction to it and how has it influenced your work on PySnooper?
  • One of the notable aspects of your work on PySnooper is the amount of attention that it garnered shortly after you published it. How has that visibility affected the long-term popularity and use of PySnooper?
  • What have been some of the most interesting, unexpected, or difficult aspects of creating, maintaining, and promoting PySnooper?
  • What do you have planned for the future of the project?

Keep In Touch

Picks

Links

Click here to read the raw transcript...
Tobias Macey
0:00:12
Hello, and welcome to podcast, the podcast about Python and the people who make it great. When you're ready to launch your next app, I want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at linode. With 200 gigabit private networking, scalable shared block storage, node balancers, and a 40 gigabit public network all controlled by a brand new API, you've got everything you need to scale up. For your tasks that need fast computation, such as training machine learning models, they just launched dedicated CPU instances. They also have a new object storage service to make storing data for your apps even easier. Go to Python podcast.com slash linode. That's l i n o d today to get a $20 credit and launch a new server and under a minute, and don't forget to thank them for their continued support. This show, having all of your logs and event data in one place makes your life easier when something breaks unless that's something is your Elasticsearch cluster because it's storing too much data. Chaos search frees you from having to worry about data retention, unexpected failures and expanding operating costs. They give you a fully managed service to search and analyze all of your logs from s3 entirely under your control all for half the cost of running your own Elasticsearch cluster or using a hosted platform. Try it out for yourself at Python podcast.com slash chaos search and don't forget to thank them for supporting the show. You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest and machine learning and data analysis. For even more opportunities to meet listen and learn from your peers you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media chronium Global intelligence, a luxio, and data Council. Upcoming events include the data or orchestration summit and data Council in New York City. Go to Python podcast.com slash conferences to learn more about these and other events and take advantage of our partner discounts to save money when you register today. Your host, as usual is Tobias Macey, and today I'm interviewing ROM Rahu about pi Snooper an alternative approach to debugging your Python projects. So Rob, can you start by introducing yourself?
Ram Rachum
0:02:23
Sure. Hi, Thomas. I'm happy to be on the show. My name is Tom. And I've been a long time a developer. I've been involved in open source for a long time. I would say I've been programming Python for 10 years. Most people know me in the patent community as the organizer of power the year, which is very, very powerful community. We meet once in a couple of months. As a meetup in the evening, where volunteers from our community give talks about the various topics, sometimes Python related and sometimes more genuine software development related topics. And over the years, I will These small companies and the companies go to startups and also the financing. In recently, I started working for a company called Blue Line. So I should only work there for four months, their company does FinTech and Israeli American startups. And so I'm slowly getting to know him a new environment. So I've been working there, it's a good place to work. If anyone here we are looking for all kinds of software development and all other kinds of positions, both in the Redwood City, California offices, and the Tel Aviv offices. So if anyone might be interested in joining blue line, feel free to send me your CV and that's my, the careers page in my email in the show notes is it's pretty good company. They closed an investment round over $100 million few weeks ago. So we're all very happy about that. Another thing I'd like to promote before I get to talking about the MMA thing is they also give the Python workshops inside business. I have a website a Python workshops that co where where I go into detail about my services, but the roof companies sometimes hi You need to come to their offices and view courses can be like three days or five days, where I teach their teams the way I prove their Python skills. Sometimes I get to teach beginners and sometimes more advanced people. I tell all the courses to their clients, their demands. So it's a side business I do if you might be interested in check on the website, python workshop.co. So that's pretty much about me.
Tobias Macey
0:04:23
And do you remember how you first got introduced to Python? Sure. A, I was
Ram Rachum
0:04:28
a college student in the Technion Haifa and I was reading Paul Graham's essays. For anyone who isn't familiar with Paul Graham. He's the founder of Y Combinator, which is the most I would say it's the most well regarded the startup accelerator. And so I was reading his essays. He is famous for having all these interesting essays on his website. I don't remember the address, but if you Google for Paul Graham essays, you'll get them that he has very interesting essays about programming and startups in finance. I was very interested in and that was 15 years ago when he was just starting Y Combinator, which was just the first batch, before everyone realized he was a genius. So many startups at Airbnb and Dropbox came out of there. So I was reading his essays. And in one of his essays he offhandedly mentioned, the Python is a good language. So I sort of made a mental note to myself, Python might be a language I would like to learn. So a few years down the line. When I was getting back into programming, I had eschewed this path of being a try. And I used to program when I was a kid, I used to program and see the sky when I was in junior high. And I was always frustrated by all the technical things that you have to remember all the details. There was there was very difficult for me. So I figured I hate being a newbie. He could be a newbie. So so annoying. So I figured, I'm going to take this one language and I'm going to know it's really weird. And I really, really hope I won't have to be a newbie language again. That was the new years ago, and my fan kind of worked, and it kind of became an expert. And I really hope I really hope I'm going to be able to keep on the same language
0:06:09
for the remainder of my career.
Tobias Macey
0:06:11
Yeah, what was the trajectory that it's on right now? I think that it's a fairly safe bet. But you can never tell in the area of technology because the face can be quite fickle.
Ram Rachum
0:06:20
Yeah, I know, lots of people falling in love with it right now.
Tobias Macey
0:06:25
And so in the process of learning Python, and using it for your day job and doing these trainings, you ran into some of the shortcomings of debugging in the Python ecosystem. So I'm wondering if you can just start by discussing some of the existing state of the art for debugging and Python, some of the shortcuts that people might make and some of the pain points that you were dealing with that led you down the road of creating pi Snooper and then we can talk a bit about what it is,
Ram Rachum
0:06:52
so I thought it was like using debuggers. Ever since I started programming, I would always do all my work in a debugger. I use a wing ID. And I know most people have never heard of it. Most people who most people use Python for their debugging. So it's basically the same kind of software. It's a good program to install. And it's the same same old kind of debugger that most languages have. And I'm very used to using a debugger. And I use it all the time. And I set breakpoints and I step through the code and I have my debugger automatically stop on exception, then I can travel up and down the stack and, and explore any kind of variables or run any kind of preparatory code, which I find is very useful for figuring out what's happening in your program. So I've done this since forever. For me, this is programming. But the when I started working with other people, especially on the side working being companies, I started, most developers don't use debuggers. And that blew my mind. And that really blew my mind. I mean, I was thinking, I'm going to see professionally purpose. We're very serious. But it turns out the They just, most of them use print statements. If they had a piece of code, and it's not doing what they think it should be doing, and they want to investigate, they add p statements or temporary log statements. And then they see the print output. And then and then they kind of have an idea of what they could do. Like they could add things technique at certain point in the code. And then they would know that that that certain point was reached. And they would add more p statements, different different parts, where they would kind of try to figure out the flow of the program, they will maybe expose a certain variable in a print statement to know what it is. I've done this in the past. We've all done this. But I was a little bit surprised that this method is so common, and the reason is so common, is because when you're working for a company, especially if it's a big corporate thing, the setup is usually very convoluted. I mean, you're working on a project that was inherited to you from someone who inherited it from someone else. It's usually a mess. record that you're writing is not running on your computer, it's running on some kind of other machine that your computer is connected to. And there is probably a complex bill crosses the chaff is your Python code around. And most people don't really know how to connect their debugger be pochamma with ID or any other program. Most people don't know how to get the debugger to work with whatever code will set up their corporate has. So most people just resort to using stateless which was very surprising to me. I'll be using p statements for debugging. I think it's both I have mixed feelings about it. I think it's both very beautiful and very part of course is is how ridiculous it is because you have to insert these print statements manually to code and then see the output and then do it then you usually realize you wanted to put the print statements somewhere else to do a back and forth feeling your project most of the time just to get the information you wanted, which to me such frustrating process. So that Sorry. But the nice part is that it always works. It doesn't matter how can we look at your Saturdays you can always print to standard out or at least write too high. And this is the reason everybody uses print statements at some point or another, because it's so really resilient talks everywhere. So I'm getting to a thought about what he was passing over. So I was thinking maybe, I mean, after many years of seeing people use principles, I was thinking, maybe I could do a compromise. Maybe I could provide a solution that is as resilient as speed statements, and as easy to use him but a little bit more powerful a debugger. So that's what I'm going to describe have someone would use it and what he provides, let's say you wanted to use by surface to devalue code. So obviously you do a pip install a price, no pain impulse buys, no. But then you go to the function which you want to understand what's going on. And you decorate it with the Heisman Trophy, dasu decorator, and what that does. basically done sort of automatic log your function as if you put a print line on each and every line of your function. And this means that when you run your code, you're going to get sort of output text of all the lines of your function, the trend, one by one. And anytime a variable got changed, you're going to get Alliance a variable exchange and its new value is recent that says, sort of instead of going back and forth, and adding p statements, sort of just automatically prints everything you might need when you can take this text output and look at it and sort of figure out lately what's going on. So that's, that's the gist of the solution.
Tobias Macey
0:11:42
Yeah, as you mentioned, print statements are easy to get started with, they work all the time. But then there's also the added issue of you need to make sure that you actually go back in and delete them so that they're not running in production and potentially leaking information into log files or into just the standard output. So that's one of the We're using log statements are a little bit preferred because you can set the, you know, set them as debug or error level. So you can have a little bit more control about when they show up. But it's still littering your code base with a lot of extra information that is only useful in a particular context and might be useful down the road to expose it to people. But like you said, there's the back and forth of Oh, I want it here. No, actually, I wanted it there. And oh, I actually needed to know what this other variable was. And so as the program evolves, those log statements and print statements become a lot less useful. So I'm wondering if you can talk through a bit now about what pious nuber is, and some of the problems that you're solving it so
Ram Rachum
0:12:35
doesn't need to be back, stronger, very advanced debugger, it aims to be acutely to do at field the hole in the market as far as the buggers are concerned. So starting to be a very, very powerful thing, just just something that passes a very specific need to what was it the rest of your question,
Tobias Macey
0:12:54
just wondering if you can describe a bit more about sort of what peis nuber is, and some of the issues with print based debugging a log based debugging that it addresses
Ram Rachum
0:13:03
in the main issues, others have to go back and forth, you have to build your project multiple times, lots of especially in the corporate, they the build process can be slow. Sometimes you can take a few minutes to build your projects. And I'm sure that some of our listeners are working in corporate environments where it takes more than a few minutes to build your project. So having to do the back and forth of building your project multiple times. That's such a drag and bicycle perhaps to avoid that, and also having to kind of guess the information that you want to expose me to write the string formatting lines, it kind of exposed this library that Bible having to do with binary, you know. So that's pretty much what the Bible says it saves you from having to do
Tobias Macey
0:13:46
so for people who are feeling the pain of dealing with these print statements and log statements as their debugging mechanism or who have been fighting with built in PDB debugger for trying to figure out how do I actually get access to it, particularly if you're running in a web request. And you don't want to use some of these debugging tools that open a socket. What's the actual workflow for getting started with pi swooper? And starting to use it for investigating problems in their code base? And what's the life cycle of a PI Snooper statement? Once you incorporate it into the code? Is this something that you would then need to go back and remove like you would with print?
Ram Rachum
0:14:22
So I start so man, yes, you will need to go back and remove it. It's famous in this regard. Now what will be the workflow for someone who wants to use by soccer, you could read the readme which is pretty short, but they tried to make it shorter. You can still buy snippets snipper import by snowcone in your code, and then add the personal production of decorator to your function. Now there are a bunch of arguments that optional arguments as to breaking the sniff decorator can get. But you don't really need all of them to get started. Just a few things I said. It is taught by super important super decorated function. Personal Fitness no is already working. As we got more arguments, you can pass the for the first argument, you can pass the filing. And then instead of writing to stand up arrow, it would write the same kind of text them to your file. Now a bunch of other useful arguments, you can say, set different debts, the default is dead equals one. And this means that only only code lines in your function are going to get traced. Now, if you were to say that because it will also go into any function that will function calls, if you were to say that equals three, it could go for us to go into any function called by function that you have to cause and you get it. So so that is one useful standing there. I think. I think that 10 different arguments there that you can modify to kind of tweak it to your needs. But the very basic usage is just a few things I said at the top is people still buy super people to buy super and decorate with their personal business. Nope.
Tobias Macey
0:15:55
So for somebody who is using pi Snooper, what are some of the types of information That it surfaces and how would you incorporate that information into the overall debugging process of identifying? What's the control flow for the logic? What are some of the branch points? How do I figure out what the values are given these different operations and understanding how to actually fix the problem that you're trying to debug
Ram Rachum
0:16:21
him most of the information that's provided is what I mentioned so far, which is a the order in which the lines ran, because every time line runs, it produces a line text dump. So you basically see the runs the lines running one by one, every time a variable gets changed, you're going to see the output the value of the new variable because a new line. So this is basically 90% of what you need to debug. Now, sometimes people would also like to call arbitrary code and because certain function and to get its value at the point, so we also have an argument for that called, watch it works basically the same as a debugger, you just put any arbitrary argument that you desire, and you're going to see its value. And its value changes, you're going to see the new value at a time, in, which is one of the arguments that we've gotten past, okay. And if you go into the base of the documentation, you're going to see that the competition both for watch and for all the other items that we have in the show documentation, just a couple of unified, potentially kicked by Uber as soon as possible.
Tobias Macey
0:17:27
And so I think that's a good opportunity to talk through a bit about how price nuber itself is actually implemented. And some of the evolution that it's gone through from when you first had the idea of it and to where it's gotten now and maybe some of the types of projects that you're testing it with to guide the direction of where it ended up. Okay,
Ram Rachum
0:17:47
so these were a few questions that one I'm gonna back into one of my one. So you asked about how festival works, right? Yes. Okay. So plasma works by using sis dot sensories system set, which is the mechanism that she bought. improvised and that's basically any kind of debugger that you've ever used using Cisco saturates who can also any kind of coverage measuring tool, but also uses system surgeries. What is the system Citrix in its pattern, a spinal function that you call it, and you pass the function into it called tracing function. And you can see Python, please use this tracing function, please for every line that runs in my code, I would like you to call this trace and function with information information such as which line ran and what kind of event happened in just regular call line or function returned when there was an exception. And any other useful information like the exception is raised or the or the value that the function return in basically any debugger or any coding any coverage measurement tool works by putting its own logic into that race function. For example, if you were writing the debugger, then some legitimate put in your face function is a is the See whether the current line is on the list of the breakpoints that you've got. And if it's on the list of breakpoints, and then you should pause the program at this point. Or if you're writing a code coverage tool, you will look at the current line that's running. And you will mark that in your database as a line of deeds run. So in the end, you can give a report of which lines right in which didn't need it. So in Facebook or my logical different, I use sister accessories. And I put my face function and my function. What it does is it checks whether this is aligned that we should be tracing, it checks whether the variables got changed from the previous line, and it outputs line thing which I ran in if any variables changed, which variables changed and what the new values are. So that's basically how it works. So sci fi from does the heavy lifting, and I just didn't do a little bit of logic around that now. Sorry, I forgot the other part of your question.
Tobias Macey
0:19:51
Just curious about some of the ways that the implementation of pi Snooper has evolved since you first began working on it and having the idea about it. And some of the types of programs that you were working on that influenced the direction that you took it. Interesting good
Ram Rachum
0:20:07
to talk about how to vote, it mostly didn't change it mostly. I mean, once I publish it, and he got very popular, we're going to talk about that some people had features and fix bugs and polish it. But mostly the basic operation stayed the same. And it wants to stayed stayed the same, just the faster and stronger and better. You know, there is one interesting thing to talk about here, and use the system saturates. And one of my contributors, Alex are we also going to talk about later, he found that there is a sort of a optimization, partially optimization, but the more low level way in Python, to control the tracing function, which is something called f underscore trace. He added it to the code and it worked and it also it allowed using a personal power as a context manager or an admin. Know How did that, and that was magic. So I googled it. And I assumed I would find information about the Python documentation. But it didn't. It wasn't some sort of magic that he found from, from inspecting a different debugger. So So after, after we published this be our sample. But why not add this to the button documentation. So other people again, know about that. So I made the PR to perform. And I've contributed to see Python in the past, but it was years ago. And recently sea bass and move to GitHub, which is a, which is a great move, in my opinion. So this was my first time contributor to see by phone on the GitHub, so I can learn the new process. If you had a bunch of bot to do things for you. It's pretty cool. And I wrote the documentation that explains what the patch tracing for other people who might want to use it, and I wrote it then we did a few back and forth with the developers there for a few weeks. And eventually, it was accepted. So now it's part of the documentation. So I'm happy to have you This resulted in a contribution to the pattern itself. Yeah, I'm gonna I'm gonna move on to your other question. You asked what kind of project are used by super on it, they determined how its evolved. The funny thing is, the first time I've actually used by super was well after I published it, and it became famous, so I didn't even use it at all. When I published it, Spotify, it's like, it's like I think I wanted and I never used them. But yeah, I just always use a debugger. And I meant for bicycle to be as a sort of a last resort tool, when you can't use the debugger. And I always use debugger, so I was good. And now, I did use it like, I don't know, maybe 10 times in the last couple of months. And it was my job. And I was to use the day when debugging scripts or my debugger ring ID. So it's very useful for these kind of times where something is convoluted and you can't, it's like, I write scripts for my ID and my ID runs them in some convoluted I don't know how to debug the scripts that like, that's my ID runs. So if I push the decorator in there, and I tell it to write out to too high, I know for a fact that I'm going to get a cookie that file. So that's what I really like about the best opportunity to get the output out no matter how difficult it is. So in in when it used for the first time, it was pretty much exactly what I wanted. And the only thing I felt was missing was that it should output the absolute path of the file every time to save a life of trend. So I added that as a quick feature, we made a new release with that new feature. But other than that it worked exactly like wanting to do so I was happy about that.
Tobias Macey
0:23:40
And so you had mentioned that some of your motivation for creating pi Snooper was to address some of the issues around using printer log based debugging and to make it easier to be able to get the full context of the application as you're trying to debug it without having to do all this back and forth. And I'm wondering if You can just talk through some of the initial goals that you had for the project. And some of them that you had to either abandon because they were technically and feasible or not useful, or maybe some of the initial assumptions that you had about how it was going to be used or how it was going to work that didn't quite play out how you had anticipated as you got further into the weeds of implementing it and using it and promoting it.
Ram Rachum
0:24:25
Well, that's it, that's gonna be a fun question to answer. I've been doing software development for a long time. I learned a long time ago that when you start a project, you have a certain vision, you imagine something about how this project is going to be. And once you start working on it, and especially once you let users use it, you have to pivot you have to understand things that you've had where easy actually difficult, and things you've had are not actually worth it. At the end of changing your project. I've can get used to it and taking it for granted. Now with this specific project, it kind of operate on exactly the way I wanted to, which I guess I guess I'm lucky about that. I mean, I mean, I heard about this idea in the shower, that'd be a cool thing to be an addict sort of vision for where it's going to be, then I believe that it was exactly what I wanted to be. So I guess that the one wanting 100 in a software development career, so I publish it. And I was very happy to get so popular. He says, I mean, I've written a bunch of open source projects in the past, and I did have one that reached his similar similar level popularity like this one, but but most of the time, it didn't happen. So I was very happy that that happened. And and the project pretty much turn out exactly how I wanted to be very happy about that.
Tobias Macey
0:25:39
And then in terms of the actual implementation, what are some of the edge cases or technical challenges that you encountered while working on it either in how Python itself operates or in how pi Snooper interacts with the program that you're debugging?
Ram Rachum
0:25:54
Sure. So I said before that I really like to use the debugger when I program the ironies. I couldn't use the debug When I was programming by snubber, why, because as I said before, both pies number and a debugger, they use the system such as function, and C Python provides. Now if you if you're using a debugger, and then it's telling Python to use its tracing function, and then you use PI super, and it overrides the tracing function set by a debugger. So the minute you start to use Passover, your debugging stops working, I mean, assuming you're debugging Passover. So that was really annoying to me. I wanted to debug myself when I was developing it, I wanted to see why it's not doing what I should be doing. And I just couldn't do it with the debugger. So I ended up using print statements, which is Yeah, pretty ironic. And I also wanted to use code coverage tools for my test. We have an extensive test suite with a very broad and I wanted to use the coverage measure measurement to see which lines are being covered. And I can do that because because coverage measurement you also use a system etc, which gets overridden by Battleborn. So what I would like to see a change I would like to see see Python is that instead of having just just one kind of One kind of forcing function that you'd be able to have an entire stack of trading function. Duty, for example, I'm writing a code coverage measurement tool, which set its own tracer. And then I use pi snover. It's going to set its own tracer on top of the previous to creaser without overriding it. So both races are going to be active at the same time. That could be a good subject for a bit. And personally, I think I think I floated the ideal path and ideas and the path and it is merely list. I mean, and there wasn't much enthusiasm, which I can understand because because people who develop debuggers and code coverage tools are really nice crowd. So there wasn't a lot of interest. But that could be a changing battle, which will make it easier to write debugging tools. But yeah, it also add complexity to the project. So I'm not entirely sure it was it would be worth it. Maybe it will be worth it to make a branch of foreign policy by phone, which will be used only to develop the binding tools for actual Bible. But yeah, that's still an issue.
Tobias Macey
0:27:55
Yeah, it's one of the challenges of having a community that's as bright Odd in scope as Python is that, as you said, for something that's so nice, where there's only a small handful of people that are doing it, that's a small percentage of the overall Python community, which is still a fair number of people, but in the context of the people who are trying to drive the language forward and ensure that it meets all the needs of everybody who's using it, it's difficult to get the focus and attention and by and necessary for being able to push forward a feature that's going to be useful for such a small group of people. And so yeah, it's interesting how that plays out in terms of open source and community management.
Ram Rachum
0:28:34
Yeah, that's right. When I was younger and more stupid, I would suggest lots of things to pass on ideas, and most of them will be stupid. And once in a while, I hit on something that was approved, but but most of the time, I would suggest things and everybody will knock him down. And it's especially going to knock my things down, which is really funny for an inexperienced developer, like you posting idea and guru says, No, this is not good because this and that and then after he says that Five more people after you say no, you thought the good idea because of this and that, and I thank you very much good already, implying that was immediate. Now you guys, you guys are so good for heavy. But I know I know that the fast winning is it was for me they were doing a good thing by making sure that everything that goes into sci fi families goes through a lot of scrutiny because it affects a lot of people. So I think they were doing the good thing, even though it was frustrating to me. But when people do make suggestions and try to improve it, I try to be as nice as I can be. Some people don't do a really great job in their peers and their issues. And they're easy to work with some people I have to correct them again and again and correct their mistakes and teach them about how to contribute to open source but also should always try to do it in the way that's most encouraging. If they do something good, I would compliment them on something good at the deed. And I would thank them for trying to contribute because I know that not everybody. Not everybody is contributed to open source and for some people it's a very frightening prospect So So I try to be as welcoming to newcomers as possible.
Tobias Macey
0:30:03
Yeah, open source. community management is definitely a challenging beast and one that has a lot of different aspects to it. And then there's also a lot of different aspects to actually contributing to an open source ecosystem where part of it might be contributing to the actual project itself in the case that you're discussing as far as proposing these peps, or wanting to implement additional functionality in the Python language, or it also might just be in terms of building on top of it, as you've done with PI Snooper and another case of that is a project that I found a little while ago called Snoop, which builds on top of your work and for what I was able to tell add some additional features in terms of user experience and maybe some additional functionality. And so I'm curious what your initial reaction was, if and when you found out about it, and maybe some of the ideas that you have incorporated back into pi Snooper itself after playing around with some of the functionality that they added and Just your overall reaction to that project and the fact that you gained enough popularity that somebody is building another layer on top of what you've done,
Ram Rachum
0:31:08
right? So it's not just somebody in the guy who makes Lucas called Alex Hall, and he's the top contributor to festival. So he implemented a lot of features. But he also had do a lot of reviews for other people with limited features and bugs. So he's been the most helpful contributor I've had. Now, there were a bunch of his features that you wanted to add that I felt were, were nice to have, but they will not work added complexity in the long term maintenance burden, and they would create now we had both back and forth about it. And he wanted, he just wanted to to met them. So he made his own fault. So I'm happy. I'm happy he made his fork. So he could do what he wants, without me being involved and telling me We can do that. Yeah, I'm a little bit unhappy that they call it Snoop. Because I think it might be confusing to people. I mean, we use by Snooper decorator skorts loop, so having his project Cosmo could cause confusion between the projects. so unhappy about that decision, and I don't have a trademark or stoops or kids reinforce that. So I'm happy that he has his fork him. I don't remember whether I marriage anything from it. It's possible that he submitted because at first got into Snoop, so I can't recall any of these. I do recall looking at the project with me and seeing that there were a few nice to have features, but a, but nothing that is really a huge difference. And I think I think there is a mistake that developers make in their really into their thing. I mean, if you read, I mean, we were both working. And I think developers kind of tend to be really into their thing and not appreciate how different the reward is. I know I'm putting it in a way that's kind of difficult to understand. But for most people who use pi snipper, they would never touch the more advanced features. Never only I mean, most people want to use code. I mean, this is something I learned when I start Welcome to be carbon back for me and you. If we use package like bicycle requests, or Django, it is very clear to us that we're using a package some roads, and it has features. It has documentation, you can read about them. And it has all kinds of things that you can use. But most people, most programmers don't really care about that. I mean, use pipe smokers because they saw some other developers by stuff, I saw the copy pasted his code and to their code until until it work. And then it worked. And they won't even bother to check the more advanced features. So people will use the more advanced features that are kind of a minority, so not have advanced features, just realized their benefit compared with the maintenance burden. Isn't that great?
Tobias Macey
0:33:43
Yeah, one of the most important and most difficult jobs of anybody who's maintaining a project is understanding when to say no to a given feature. One of my favorite sayings is that open source is free as in puppy where somebody can give you a puppy for free but then you're on the hook to actually take care of it and Make sure that it's well fed and well maintained throughout the entire rest of its life.
Ram Rachum
0:34:03
That's a nice metaphor never had that one before.
Tobias Macey
0:34:05
So as you alluded to before, a couple of times, one of the notable aspects of your work on pi Snooper is that shortly after you launched it, it garnered a lot of attention broadly within various communities. And so I'm wondering if you can just talk about some of the strategy that you used to gain that level of popularity and a little bit about how that visibility has affected the long term popularity and use of pi Snooper and maybe some of the future direction of where you're planning on taking it,
Ram Rachum
0:34:37
right. So I'm guessing that too many of our listeners might be interested in writing their own open source projects and maybe maybe get him to be very popular. So I'm going to share what I did and what I think about what contributed festivals popularity. So the first thing that's very important, you can have a readme 3d explains what your project is about and what kind of needs it's fulfilling how it's supposed to have it would be me good tagline for free report. And I understand this is not a marketing effort, you now have to convince the user in like 20 seconds and 30 seconds, that you've got something interesting, showing something cool. When someone can I mentioned this in the previous census, people are less interested in your project, when you are in this means that if you if they're going into your project college, they have like 20 seconds or 30 seconds of attention. They are bored. They want to do something guys, but the thinking maybe maybe you could entertain them. So it's really it's really important to get your message across. So spy stuff I kind of described, the tagline was by super liberal use print for debugging, again, which I think kind of hit it home. I mean, people can relate to that. And it kind of made me the explanation is when you're you're trying to debug and you want to know the values of the variables, but getting any buggers to disagree, because you will copy the vironment kind of described the exact scenario that I've seen play out 1000 times and then and then I had sort of a quick start example of how to use it instead of making to do documentation and having fun I'm going to go through a complicated tutorial. So it's really important to me to get that reading straight, so people could be interested to learn more. So that's one aspect. The other aspect, you got to do sort of a launch, I mean, when you make it public, should they opposed to hacking us and post to Reddit and post to wherever as you can think of, you should get a few of your friends to avoid it in the first few minutes for you, when you're posting supporting, I would say this is the only kind of dark path and it's kind of cheating, but you only like three or four people to get to the front page to for a few minutes. And once you've got that you're not really cheating anymore. Because if people like it on the fan page that they will upload it if they don't they want at that point, it depends on how likable your project is with PI super eager to get more points on Hacker News to them to be number one there. So lots of people started on GitHub. And once people started on GitHub, it got to the trending page on GitHub, and then more people found it through there and more people started and then get to be number one on this on the trending page of get up and then it really exploded. I got lots of stuff As contributors, I get the I think at this point I'm almost reaching 13,000 stars, which is insanely unproportional to do to the project because I think like projects that see Python or or NumPy have like, I don't know, I think 30,000 stars. So obviously my project is not half as complicated or half of whatever is the bottom. So I guess I was just just lucky to get all that attention. You asked how it influenced the project. Well, I got lots of country contributors that have been easy to work with like like Alex whole with all super helpful pull requests and issues. That's what people like submitted issues who started deals work for them and obviously they obviously know no one ever yelled at them before for not including information either issues. So I would I would get just like just random. Like for example, today or I think yesterday just someone just said, Hey, I got this exception using pipe number. And you just just pulling in with no context about what he was doing. And Alex and I had to kind of question even Peter gave us the right information so that people can submit issues in a way that has no information in it. Also, in the last few years, lots of people in Asia got into open source. So there's lots of people who just just great issues in Chinese, just riding in Chinese without a thought about whether I speak Chinese and I don't. So So that's, that's pretty funny. You know, as long as I'm speaking about these kind of issues, when I started doing open source development, mailing lists were a big thing. Like every project had a millions. And I personally hated it, because it was weird that you had to subscribe to something to ask question. So now there is no Middle East, or is there isn't one for most projects. So when people just asked questions on the board, but they but they were having fun, and they've got lots of contributors, I think I've gotten around 20 computers. So the popularity improved, improved the project called contribute gave me the project. So I'm very happy about how it turned out.
Tobias Macey
0:39:04
And overall, what have been some of the most interesting or unexpected or challenging aspects of creating and maintaining and promoting pi Snooper and maybe some of the opportunities that you've gained as a result of its popularity?
Ram Rachum
0:39:18
Yeah, I had fun speaking about it in a few confidence with two guys. I think I mentioned most of the interesting unexpected things above. And I had to say, but I had fun speaking about it in a few conferences in Python, as well, and in Vikon, in the Czech Republic, and also in your Python in Basel, Switzerland. So that's been fun. And that's pretty much it. I try to you know, I've been very tempted to maybe add more features and more fancy stuff, but I think that the the strength of Cooper is that it's, it's so small and unambitious. So I try to keep it small because if I if I start adding more ambitious features to it, I think it is kind of ruined the project. So so that's pretty much for me. I mean, as far as as far as I'm concerned I don't think I'm interested with regarding this project is a long term maintenance and reliability, make sure it fits with the new iPhone version. Once in a while someone submits a feature request that it actually happened a week or two ago. And they and I work on it, but mostly just the budget is fine. And I tend to change it to any major features at any point.
Tobias Macey
0:40:27
Are there any other aspects of your work on pi, Snooper or the state of the art in debugging within the Python ecosystem or anything else related to your work that we didn't discuss yet that you'd like to cover before we close out the show?
Ram Rachum
0:40:38
Sure. And one thing I say, which is actually not my work in you've mentioned PDB, early that isn't I hate PDB with a passion, I hate it so much. And I remember using it like one once or twice when I was starting my career. And I wanted to use q as a viable name, just because sometimes you use random letters as a viable name when I'm debugging What I talked to you with immunity x, and I was so angry and I never used PDD. Again, I was very angry about that. But I learned there is a different program called p UDP, which is sort of similar to PDB, as in it, as in. It's a debugger that you use in a shell. But it has a sort of graphical interface in the shell, which is kind of reminds you of the goal and see the face of the past. It's really cool. It's basically like pdv, except it actually shows you record it has sort of a graphical interface, but it's still in the shell and a little quiet, gooey. It's like a Windows or whatever. So that's a pretty fun tune once in a while when I get shell access to something that I think I think everybody should at least give it a try to get it to be useful to me, you know, so many times.
Tobias Macey
0:41:45
Yeah, there's another project along the same lines, that's called PDB plus plus or PDB PP that will transparently work in place anywhere you use PDB for the built in module and just adds a lot of niceties in terms of center Highlighting and better visibility of the code that you're currently within the context of, but it's not quite as full featured as p udb. So it's sort of a step in between. But there is definitely a lot of value to be had in terms of understanding how to use a traditional debugger for being able to step through the code, and then being able to have things like your work on pi snipper available for being able to just quickly get a good overview of what is the overall logic flow and the variables that are there. So understanding what all is in the tool chest and when it's appropriate to use it is definitely an important skill for any engineer. Right? All right. So for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And with that, I'll move this into the pics and this week I'm going to choose pi con us it's the largest pi con event. It's the one that I've been to a number of times and plan to be at again for the spring so they actually have the call for proposals is still open. And until December 20, if you're listening to this before then and registration has been open, so I hope to see a lot of people there again. And with that, I'll pass it to you. Ron, do you have any pics this week?
Ram Rachum
0:43:10
Actually, something about yoky. I submitted a proposal to us, we don't get accepted about the recording and music synthesizer, some really fun talk. So I really hope they can accept and so I hope I'm going to see you at Facebook. Regarding my ticket, I would like to talk about something that is completely unrelated to programming, or I think it's a fun thing to Peter Shinola. It's called nonviolent communication service school of thought about expressing your emotions and interacting with other people. Around the 15 years ago, a friend of mine sent me a YouTube video of this guy spending a lot of time communication, what was it about and he kind of blew my mind really changed the way I interact with people. High quality in one sentence about expressing your emotion in a way that is a completely free of judgment and separated from your paths to the learning how to listen to your emotions, way to separate from your thoughts. You get to learn a lot of things when you do that. So I'm going to link to the YouTube video in the show notes, and hopefully it's gonna blow your minds as much as mine.
Tobias Macey
0:44:19
Alright, right. Well, thank you very much for taking the time today to join me and discuss your work on pi Snooper and some of your experience of helping to grow the community around it and see your experience of promoting and marketing and open source project. So it's definitely an interesting tool and one that I am sure I'll have cause to use in the near future. So thank you for all of your efforts on that front end. I hope you enjoy the rest of your day.
Ram Rachum
0:44:43
Thank you very much for having me on device. It's been a pleasure.
Tobias Macey
0:44:48
Thank you for listening. Don't forget to check out our other show the data engineering podcast at data engineering podcast.com for the latest on modern data management and visit the site of Python podcasts. com to subscribe to the show, sign up for the mailing list and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcast in a.com with your story. To help other people find the show, please leave a review on iTunes and tell your friends and co workers
Liked it? Take a second to support Podcast.__init__ on Patreon!