Zulip Chat with Tim Abbott

Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great.

I would like to thank everyone who supports us on Patreon. Your contributions help to make the show sustainable.

When you're ready to launch your next project, you'll need somewhere to deploy it, so you should check out linode at ww

w.podcastinnit.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or experimenting with something that you hear about on the show.

You can visit the site at www.podcastinit.com

to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. To help other people find the show, please leave a review on Itunes or Google Play Music, tell your friends and coworkers, and share it on social media. Your host as usual is Tobias Macy, and today I'm interviewing Tim Abbott about Zulip, a powerful open source group chat. So, Tim, could you please introduce yourself? I'm currently the the lead developer of Zulip, which is, you know,

you know, leading open source, group chat software. We're actually by sort of a contributor volume, the you know, both in terms of total number of people and just how much activity is going in the the leading open source group chat, platform,

today.

And,

you know, I sort of work on basically every aspect of the project, you know, from the, you know, the web app to the mobile apps to the integrations,

you know, to the sort of core tooling and so on. A little bit of sort of background, you know, I've previously sort of spent a lot of time doing startups. So I was, the CTO of Ksplice, which was

technology for, updating and running Linux kernel without rebooting. You know, we, you know, sort of got that technology working basically flawlessly for turning, like, a Linux source code patch,

as a sort of the input into, you know, our bootless kernel update for that, source code patch in a, you know, sort of completely automated fashion and ended up running a subscription service with, like, 100 of thousands of servers being updated with it. So that's sort of 1 piece of my background. The other, sort of piece of it is, you know, sort of ZULIP's own backstory. So I'll but I'll get into that when we we talk more about Zulip, I guess.

And how did you first get introduced to Python?

Sure. Yeah. So it's,

you know, there's sort of 2 versions of that story. So I think my first experience with Python was was actually in high school. I was taking,

an AI class, which, I guess the the sort of you know, that class has historically been a class where, you know, 1 used, you know, scheme and did, you know, algorithms like a star and and and that kind of, you know, genetic algorithms and neural networks and that kind of stuff.

But, you know, we had this, you know, new teacher of his who is really,

and was really into board games. And so he decided that that what people are gonna do for their sort of final final projects is write AIs for the

game of risk. And, he really, you know, sort of that sounds like maybe a great concept for project. And in fact, it was, but he had no intention of actually building a greater to actually have them

play against each other who is just gonna,

you know, just gonna have people write it and grade them based on how interesting their algorithm was or something. So, you know, a friend of mine, you know, ended up deciding to write write a grader for this, and he wrote this sort of framework in Python, which, you know, you could sort of watch it play, and and there'd be sort of an API where your program could

could play a rest. So that was sort of my my first experience sort of interacting with Python.

And it was just pretty cool, like, how he was sort of able to throw together this this whole system, you know, pretty quickly with the the Python, graphics libraries and so on. But I sort of really first got into Python

actually when I was working on ksplice. So, you know, I sort of had a lot of experience with Perl,

before that,

but I just always found it, like, kind of frustrating to do something complicated with Perl because rather than writing a ton of code yourself or having to grab all these sort through tons of sort of low quality modules on CPAN. And

it ended up just being a really great experience

for Caseplay. We built all of our sort of automation infrastructure,

the sort of desktop client you could run on your Ubuntu machine that

sort of worked like Update Manager where, you know, it'd tell you, oh, if you got some newer bootless updates installed and you just click a button and installs them and it'll show you like, oh, here's the CD numbers that you just fixed and that kind of stuff and the web application. We just found it like a sort of really nice platform that you could use to

sort of build, you know, an entire,

set of applications,

anything from sort of like our sort of automation to our, you know, sort of the whole thing, and just have sort of 1 language that you needed to know in order to do any of the

sort of work at our company.

And, I'm sure that led nicely into your work with Zulip. So I'm wondering if you can dig a bit into what Zulip is and what was the initial inspiration

for creating it.

Yeah. So the Zulip is probably 1 of the the more fascinating histories for for sort of open source projects out there. So I guess, like, what is Zulip? So Zulip is, you know, sort of several things. It's a it's a full featured, you know, modern group chat,

you know, application, you know, and and by full featured, I mean, it sort of has everything. There's, you know, a server,

there's, you know, a sort of web application, which is sort of the way that, you know, most people might start out using it. There's, you know, desktop apps implemented in Electron.

We have, mobile apps, which we have recently switched to to be on React Native, which has been a great experience so far.

You know, there's integrations with, you know, every version control system, you know, every, you know, popular,

issue tracker,

dozens and dozens of sort of native integrations, 100 more that you can do through things like IFTTT,

if this and that, or

Zapier.

You know, it's got, you know, an API, there's documentation, you know, there's sort of all these different different sort of pieces. You know, second, like, ZULIP is is, completely open source. So,

you know, many of the other sort of open source group chat things that you'll you'll see around, you know, they they're actually sort of open core

and really, like, don't satisfy the Debian Free software guidelines or whatever. Zillow is really about being 100% open source. We care a lot about having a vibrant open source community with people from a wide range of backgrounds. And we've been very successful in building that. Over 300 people have contributed code to the Zulip server. Hundreds of commits go into Zulip every day. Our sort of usual release,

when you do a big, big release every few months, has like 150 distinct people who have contributed code.

It's a pretty wide open source project.

Sort of finally, it's

really optimized for productivity.

So Zulip has a

significantly different model from a lot of, you know, sort of other open source or or not, group chat tools like Slack or or IRC or whatever. You know? So all of those have sort of the

model where you have, you know, a channel, and it's just a time sorted sequence of messages, and there's sort of no no organization in there. And this could be, you know, pretty problematic if you just wanna be talking about multiple things with the same audience at the same time. You know? Because fundamentally, like, if if there's a conversation already happening, you sort of have to decide, like, if you want to bring up something just totally unrelated, like, do do you wait until that conversation finishes, or or or do you

do you just sort of mention your thing and hope somebody replies to it, or, you know, it may actually just cause that other conversation it's just really hard to have talk about multiple things at the same time at the same group of people.

And this has been a frustration of mine, you know, in IRC, you know, going back, you know, for a very long time. And the same issue is true in this precedent in Slack. It is just sort of fundamentals of the model. So the the way that sort of ZULIP does things is sort of the same solution that that email uses for this problem. So you you have,

basically, a subject line, you know,

on a message, and usually it's just a word or 2 because you don't need a lot of of context in chat. You know, when you send a message, it's all up, like, 97% of the time just reply and you don't need to think about this. But if you wanna bring up something new, you you send a topic. And then the whole sort of user experience is built around making it really easy to review sort of all of the different conversations that have happened while you were away from your computer sort of efficiently in this sort of 1 thread at a time sort of organized fashion. And in the sense that making it, like, you know, much more efficient to to catch up on on a lot of traffic. But it it also sort of changes the way that an organization can collaborate because, you can sort of expect people to read stuff that's important to them. You can, you know,

7 conversations that happened. And you're also just much more likely if you ask a question for somebody to reply because it's not just sort of like in between the 37 message thread about cats and the 27 message thread about, you know, politics or whatever. Right? Like, it's you know, each 1 has sort of its its own space.

So that's sort of, I guess, the 3 sort of pieces of of what Zillow is. You know, it's a full featured modern group chat tool. It's completely open source, and it has sort of a different model that, you know, our users find, make them a lot more efficient at getting things done, in chat.

And how did it first get started?

Yeah. So the the backstory is sort of interesting. So the origin of Zillow actually comes out of this system,

at MIT called, Zephyr, which was an,

originally built as a printer notification protocol in the late eighties. So the the protocol is terrible. But the the sort of it had those sort of same topics feature,

just sort of a detailed how the protocol worked. Like, the the channel would be, like, printer and the, the the topic would be, you know, what the name of the printer is or or something like that. And the the sort of over time at MIT, this the system was sort of the dominant way that people did, chat. People built some very interesting clients around. There was this really cool thing called Barn Owl that was a, like, a Curses client that you'd run-in a screen session. But it just had this fantastic user experience where you could, you know, literally be be sending and reading, you know, hundreds or thousands of messages every day with your your friends on the MIT campus. You know, maybe trying to get projects done, but also just, you know, recreationally.

You know, you could sort of manage all that efficiently,

you know, in this in this tool. So we actually used it when starting Caseplay as

just because we found that it was a much better experience, for getting stuff done. And that that worked out okay for ksplice,

because, you know, if you're even if you're, like, on the sales team at a company that's doing rebootless kernel updates, like, it's totally cool to be using your curses client in a screen session. Like, you you sort of have to know enough about general Linux system

systemsy stuff in order to answer people's, you know, questions about how how does it interact with Kron or is it an agent or whatever, like, funny questions people people like to ask. And, like, I guess, like, you know, we we sort of integrated that system with with how things work. So anytime anybody got an error trying to install a Caseplay update anywhere in the world, we would get, you know, if there's a new error, a chat notification,

you know, in in our our Zephyr setup, and we could just real time debug in that thread. So, you know, and that'd be sort of 1 place. And then once we figured it out, you know, we could then, like mention the people on the sales team and then they'd be able to email the, you know, customer basically, let them know that we've already fixed the problem probably before the customer even got around to emailing us. And it was just like

that type of integration,

you know, where it was just an amazing experience

for getting stuff done and was just really flexible. You know, we integrate all of our processes in this way. And when, you know, we talked to a lot of our friends who are doing technology projects and they were pretty jealous of our setup, you know, and we were sort of thinking just like for whatever we did next, we'd want to have a similar setup. But, like, you know, basically for any other product that we could be building,

like, you know, it probably wasn't gonna be a great plan to have, like, our internal communication tool, the Acurses client running in a screen session. You know? Like, it's just sort of ridiculous.

So sort of that our inspiration was like, you know, maybe the the the next thing that we should build, you know, basically, the the founders of the original ZULP company were the same as the Caseplace founders,

was to build a,

a sort of a modern group chat tool that 1 could feel good about that had this model. So that's sort of how Zulub got started. And, you know, we we sort of, you know, built that product for for the original ZULIP company, I guess, for about a year and a half. And it was doing pretty well. Like, you know, there were,

you know, thousands of users that were excited about it. And then, you know, sort of we ended up getting this, unexpected acquisition offer by by Dropbox.

And,

you know, after some sort of thinking about what we wanted to do, we ended up joining Dropbox. So I I spent about 2 years at Dropbox.

And my my role there actually ended up having nothing to do with with Zillow. So I I I sort of did some stuff on there, stuff related to Zulips for a few months, but then I ended up running the 3, 000, 000 line Python code base at Dropbox as sort of like the

architect of their platform.

And, a lot of interesting stories from that that experience and actually a lot of things that I learned about how to, have a large number of engineers collaborate

on a project. But anyway, I guess this is sort of digressed into to to sort of my story. But, anyway, so I guess the story for Zulip there is that, you know, Zulip, InterZulip was acquired by Dropbox, and and Dropbox didn't didn't work on it, at all. So, that that lasted for about a year and a half. And 1 of the sort of surprising things is that, like, basically, none of the users stopped using Zillip of our

beta users. It's pretty rare.

Like, if a project gets acquired and, like, it's still running, but it's clear that nobody's working on it, like, usually all the users go away and switch to something else, especially given how much press Slack was getting as, like, the coolest thing ever, you know, at the time. And, actually, some of them did switch to Slack and then switched back because they decided that even though Zulip was unmaintained and the build apps didn't work and because they'd just, like, randomly broken at some point during that year and a half, like, they still preferred using Zulip to to Slack. So that was sort of 1 of the pieces of data that suggested to me that we should try to make sure that people can keep using ZULIP because people people really appreciated it. And so I ended up, you know, sort of working out this plan with Dropbox where they were okay with being open source because they weren't going to do anything with it, but we needed to do work to open source it because I don't know if, you know, if you you know, any sort of code base that's, you know, tens of 1, 000 of dollars of code, you know, you had some customers, you probably put some customer names and some comments or some commit messages. You probably have, like, some, you know, secret values that you put in the code, you know, maybe a little sloppily, you know, whatever. And you don't really want to, like, release a thing like that, especially when the server is still running, you know, in production. So we ended up, like, having a plan where where 10 Zulu users came to Dropbox Hack Week for for a week. And I sort of, like, coordinated the effort, and we got all the work done to make it possible to open source ZULIP and have there be, like, an easy way to install it both in production and and, you know, for a development environment to to sort of modify it. That resulted in ZULIP being open sourced,

about a year and a half ago. And, you know, that post is probably the, you know, announcing that it was open sourced, but it's probably the most popular thing I've ever written. I think, like, 200, 000 people read that blog post saying that ZULIP was open sourced. Basically, like, you know, after that point, I sort of was maintaining the Zillip open source project, you know, sort of nights and weekends, like, 5 or 10 hours a week because, like, my, like, Dropbox wasn't working. And my main job at Dropbox was running our 3 main line Python code base. I ended up leaving Dropbox about a year ago, and I've sort of been focusing on Zillow as what I spend my time on since then. And it's been sort of amazing to see the project go from, you know, basically just me and my my spare time to, you know, literally hundreds of people,

working on it. And, you know, I think I counted a few weeks ago and there was about 30 people who are spending their summer this summer working on Zulip, which is is kind of incredible. Yeah. So I guess that's sort of, like, how how Zulip, you know, got from where it began to to today sort of at a high level.

And just briefly, where does the name come from? There are a few possible,

sort of,

guesses that people can make based on the sort of sound and word roots, but I, I'm curious where it actually originated.

Yeah. It's, it's it's, Tulip with a with a z.

The the the backstory for that is sort of, you know, this is probably an experience that a lot of people who've who've worked on a a startup,

you know, will appreciate is you need to name your your startup. The problem is that, like, all the the really good names, like, like, you really wanna have the dotcom domain for the name of your startup, and all of the really good names are taken.

Or, you know, if you wanted to buy the dotcom domain, there's some, like, squatter who'll tell you that it'll cost a $120, 000 or whatever, and probably you could negotiate it down to down to 30 or something. But, like, it's really, you know, pretty pretty challenging. And And so, you know, we basically did a brainstorming process where we tried to think of, you know, sort of everything that had the properties you wanted to name that it's easily

spellable, easily pronounceable, you know, sort of in both directions. Like, if you pronounce it, you know, somebody else will be able to spell it and vice versa and tested them on these criteria. And we used all sorts of algorithms for generating them, including taking words and replacing letters and all sorts of other fun stuff. And, you know, this was the the best thing that we were able to to get based on that process. You know, 5 letter domain is is pretty good. But, yeah. I don't know. I think, you know, naming projects is is 1 of the great, challenges in

in, in the world. And I think it's only gonna get harder as more and more good names, you know, are a duplicate with some other popular

Yeah. Exactly. The the the 2 hard problems in computer science, cache invalidation, naming things, and off by 1 errors. Exactly. Yep.

And so,

you're saying that after the project got acquired by Dropbox, that for the most part, Dropbox wasn't doing any work on it. Was this being used internally at Dropbox at all, or was it just was it more of an acquihire where they wanted to purchase the Zulub company in order to, you know, bring you guys on board as, engineers for their project or originally have plans to make use of the Zulip platform for their own purposes?

Yeah. So that Dropbox actually did end up using, 1 component of of Zuluq for some of their collaboration projects, you know, sort of a piece of our server back end. But I think, like, their intent was,

you know, for us to build the sort of collaboration roadmap for Dropbox, you know, whatever thing that they were gonna do in that space, whether it was a a thing like, you know, Zulip or Slack or or a thing, you know, that's sort of totally different from that. What sort of ended up happening, you know, is we we sort of got to Dropbox, and our determination was that Dropbox's resources were just allocated

towards the wrong things. Like, the main Dropbox product

had just a lot fewer people working on it than anybody would have guessed at the time. We had 250 engineers, and a lot of those were working on infrastructure and scaling and stuff like that. But the the sort of team working on the core desktop client, you know, the thing that you install on your machine that is what people think of as Dropbox, you know, across Mac, Linux, and Windows, like, that whole team was, like, 10 people. And at the same time, the team working on, like, the Dropbox website and Dropbox for business and sort of like the stuff that sort of I thought of as as really the ways that were sort of the simplest plan for Dropbox to be sort of commercially successful was also like a pretty small team of, like, 10 ish people. And there are just tons of of sort of product initiatives working on new things,

some of much have have come out like Carousel. You know, there's a team working in Mailbox,

you know, things like Dropbox Paper.

And we didn't really feel like what Dropbox needed was another team working on, you know, like, another sort of product. We we thought that, like, they needed more sort of top people working on the main Dropbox product. And I think that was the right decision because, like, 6 months after we joined Dropbox, like, the Zulip founders were actually running

the main

Dropbox product, you know, across, like, all platforms, you know, mobile, you know, web, desktop. You know? And, you know, that was a a great experience for us, and I think really was quite valuable for Dropbox, far more so than I think, you know, what could have been possible working on, you know, sort of ZULIP at Dropbox in terms of value for for Dropbox. But it was also, you know, I think a great experience for for me to sort of, you know, be working on sort of a code base that, you know, was that huge that, you know, I didn't have a lot of prior context on. I'm trying to make it, you know, more productive. You know, 1 of the things that that we sort of found is, like, I was, you know,

appalled as to how hard it was to sort of get work done on Dropbox's main, Python code called the meta server. You know, and we we sort of ended up doing a huge amount of work to to try and fix a lot of the systemic

problems there, which resulted in, you know, some popular open source projects that sort of came out of that, like the the Mypy,

stack type system for Python. It's basically funded as part of that initiative. But 1 of the things that was sort of amazing for me is that, you know, when we met up with sort of my peers at places like, you know, Airbnb or Twitter, you know, their situations were far worse than the drop boxes. So actually, I think despite the fact that, you know,

fact that, you know, sort of the developer experience at Dropbox, at least when I arrived, was far worse than anything I experienced before. You know? And I think in part that's basically that I was spoiled by by building my own tool chains. And,

you know, the the the reality is that basically all of these, you know, famous tech company unicorns are, you know, sort of the internal tooling productivity situation is usually a kind of a disaster.

Yeah. Having worked at a number of startups, it's definitely interesting how much of a front you can put on where you can make people believe that you have a much larger team than you actually do just because of the number of different solutions that you can just use as a SaaS offering to accelerate your capabilities and, expand the productivity of your

sort of

crushed under tech load of technical debt and, you know, not really be able to dig

yourself

under tech load of technical debt and, you know, not really be able to dig yourself out without some very concerted effort by the engineers on staff. Yeah. I think it's a deeper problem that I think, like, my experience is that the the sort of engineering managers at most companies like or product management,

that there sort of ends up being a shift that happens when a company gets to be around 50 ish people where

sort of usually there's sort of some top down allocation of where all the resources go that gets pretty pretty forceful.

And, you know, if you're sort of a team of 3 people working on some stuff, like, you're probably gonna, you know, fix the stuff that the problems that make it hard to get your work done because it's just gonna be frustrating, and you just feel like I'm gonna make my own time more efficient if I do it. But once once you have a team of hundreds of people working on something, there's sort of this diffusion of responsibility thing where it's not, like, like, clear sort of who's supposed to do, you know, the what's probably, like, 10% of the total work, if if not more. Maybe it's 30% of the total work of, like, making it efficient to be productive

on a platform. And I think, like, you know, most product managers don't understand that, and and many engineering managers don't understand that. And so it ends up being the case all the resources get sort of tight deadlines for doing specific

user facing

results, basically.

And,

you know, the the sort of spare time that individual engineers have is not sufficient, you know, to sort of fix the systemic problems. And and you start to you start to get really buried under these,

yeah, piling up little little piles of technical debt that that sort of nobody feels like they they're in a position to deal with,

just in terms of like like, they might even know how to fix it, but they don't have the sort of authorization to work on it, you know? But I that there's also a phase of, you know, a lot of companies, you know, sort of started out as a startup 10 years ago and, you know, what what things you were supposed to use 10 years ago are just totally different now. There's just, like, a ton of work to keep refactoring your your sort of infrastructure to be reasonable

in in the current year and not reasonable

5 years ago

when your choices were different.

And speaking of the sort of

capabilities and design choices in your application, I'm wondering if you can dig a bit into how Zulip itself is implemented

and particularly

the fact that

it's a real time chat system. And most of the time, when people are thinking real time, they're probably thinking Node. Js or some other WebSocket

communication capability. But my understanding is that the bulk of Zulip is actually implemented in Django. So wondering if you can describe

the architecture and the design of the application that allows you to be able to use Django while still having that real time communication capability layer.

Yeah. Sure. So I think this is actually a common confusion, I think, that that people have when they think about real time communication. So there there is, like,

an element of real time communication that's important in a chat app. You know, you need to have

an efficient server to client push system to notify the client, you know, that there's a new message and maybe what its content is so you don't need next round trip and and so on. But,

like, that system only needs to do that thing. That's the only thing which is sort of real time essential. You know, everything else is is just, you know, you need to do AJAX requests and, you know, have them be responsive. And so we we actually have a component for doing that server to client push, which is not done in Django. It's done in Tornado.

But the whole thing is probably 2, 000 lines of code including tests. And, you know, we haven't really had to modify since 2013 because it it just has a well defined interfaces. You know? So there's basically like the the sort of structure of the architecture is that Django, you know, when it wants to send in a, you know, a real time push event to the client, you know, just,

you know, does a request over to Tornado to let it know to do that. And, you know, that's that's basically it. But there's a lot of subtle details in doing that right. And, if if anybody's interested, we actually have a quite detailed set of documentation in the on the Zulip Read the Docs that explains sort of the whole design around how our tornado system works and how we avoid race conditions and so on. But it's probably not not something I can effectively get into, you know, in this interview. But, yeah, I guess, like, sort the high level architecture is, you know, we have, you know, sort of pretty standard web application set up. We've we've got a Django, you know, you know, thing using uWSGI,

that text to, you know, a Postgres database.

And, you know, we also use memcached for caching on Redis for some slightly more complicated caches. And we use RabbitMQ for for enqueuing,

you know, events to be processed, you know, later, either for actually some communication with the tornado process, but but primarily for just sort of deferring work, you know, so that we can so actually 1 of the things that is important for for sort of, you know, a good real time communication is you want to have, like, your request to send a message, you know, that needs to be processed very quickly. So we do sort of have a model where the stuff that needs to happen immediately, you know, happens immediately. That takes like tens of milliseconds, you know, which is basically instantaneous,

for server side. You know, your your performance can be dominated by network connectivity when when when you're at that sort of level. And then the the sort of, you know, sort of the rest of it, you know, you should just write in whatever framework is gonna make it most efficient to implement all the features that you need to have. And, you know, I would argue that that for a real time chat application, actually, the the sort of big challenge in 2017 is surface area. Now there's just literally hundreds of features that that, you know, sort of people expect as sort of table stakes for just being a monitored real time group chat application. And on top of that, there's, you know, 100 more sort of

additional improvements that 1 could make that, you know, we would ideally like to have in Zulip and, you know, have been building over the last several years.

Yeah. And in terms of the architecture that you've got as well, 1 thing that I was

surprised and pleased by when I was going through the documentation is the fact that for your full text search, you're not immediately jumping straight to Elasticsearch and you're actually leveraging the capabilities of post grads for being able to do that full text indexing and do the searching so that you can reduce the number of external dependencies that you have to run to be able to, deploy the server itself.

Yeah. Using the Postgres full text search, I think, is 1 of the best decisions that we made architecturally because, like, it allows us to do, you know, a hugely expansive range of of different types of filters, you know, as to what what you're searching for, because you just translate them into the appropriate, you know, SQL thing with with no issues with synchronicity. Like, we don't have to worry about passing a feed of data over to something like Elasticsearch,

you know, to update, you know, its index. You know, every time that something changes, you just edit the rows in the database, and you're done. And there's you know, our performance is, you know, if you you sort of ask people who use search, I think our our search performance is probably the best of, you know, any of the,

real time chat applications that that I've seen. Like, it's it's really fast, and it's just because, you know, if you have the right database indexes,

like, Postgres is is fantastic. Actually, that that sort of reminds me of a little bit of our story. So we we actually originally, Zulip was built on top of MySQL. And we actually migrated from MySQL to migrated from MySQL to Postgres, like, you know, a year after starting the company, which is is not a, you know, pleasant migration. But, we we found that, like, using MySQL, you know, we were

spending all this time both working around

stupid MySQL defaults like it's, you know, Swedish case insensitive

default collation and, you know, sort of but but also just sort of from query planning perspective, we were doing all these performance optimizations, you know, where we need to worry about the query plans on. And, when we switched to Postgres, basically everything, you know, that had been fast with MySQL got faster, and we could throw out all of the, you know, sort of custom query plan

fiddling stuff that we'd done and just use the Django ORM for

98%

of database queries that we need to make and it just worked. So, you know, I think Postgres is definitely the

also 1 of the technology choices that I feel really good about. Like, I think it's saved us

countless hours of of wasted effort tuning to MySQL that is just completely unnecessary with Postgres. Yeah. It's definitely funny how quickly people will jump immediately to using Elasticsearch just because of the sort of cargo cult mentality where if you, you know, every everybody thinks, you know, you wanna have search, you have to have an external system to do that because you can't do it in your database

for x reason, without actually necessarily digging deep into the capabilities of the tools that they're already taking advantage of.

Yeah. I feel like that's that's a pattern that, you know, I don't know. Like, I guess 1 of the things I find really interesting is is the microservice

trend, you know, where sort of the idea is that you should split your your big service into, like, 50 smaller things that all interact. And it's 1 of those things that, like, is a good idea if you're at Google because your problem with your Google is that you literally have, like, many thousands of engineers working on things. And you need to figure out how to, like, create interfaces between this team of 30 engineers and this other team of 20 engineers that, like, you know, are

just easier to manage, like, organizationally.

But, you know, know, for the vast majority of projects, like, you just end up creating a bunch of overhead for yourself because, like, in order to change something, you don't have to, like, make a change in both of these services. And you would have been just as happy from a code perspective just making it, like, a library, you know, or like a Python module that has, like, a well defined interface for almost any goal that you would have.

You know? So anyway, that that's sort of been interesting. Like, I think that there are applications or microservices can be really important, but I feel like it's sort of it's an idea that has far more reach,

in terms of people wanting to

to grab onto it and and try to use it in their product project than, than it's actual utility. And I think, yeah, the Elasticsearch thing is a good example of that. And what has been some of the most challenging aspects of building,

maintaining, and growing the Zulip project? Yeah. So I I think, you know, probably technically the biggest challenge is really just the issue of surface area. Like, you know, I sort of mentioned, like, Zulip has a, you know, a Python server. We've got, like, the the sort of web application on top of that. You know, there's, you know, sort of all the HTML

documentation and and landing, you know, pages and login pages and and stuff like that. You know, we have literally dozens of built in integrations that are included in the Zillow, you know, codebase,

etcetera. And all of those have documentation including screenshots explaining, like, here's how you go into the GitLab UI and, like, configure GitLab to send us notifications

into Zulub, which is really, like, arguably part of their documentation because it's really just like how do you create an outgoing webhook, you know, in GitLab. But,

you know, we we sort of feel that, you know, we should explain you should only need to go to 1 place to learn how to, like, set your thing up and and so that's important to us. So anyway, like, that's sort of been a challenge which we've been struggling with because, you know, I think it's you don't wanna end up with your project having a 1, 000, 000 lines of code because it it becomes unmaintainable.

And, you know, I feel pretty good about how we're doing on things like that integration issue. So if you wanna write an integration,

you know, you know, that's an incoming webhook integration in Zulip, like, you literally have to write, like, 10 lines of formatting code. We we have sort of a a process that we document for how you can grab some test fixtures or basically like the API you know, the the format basically of those outgoing webhook messages that, get sent out by, you know, whatever the product is. And, you know, there's sort of a markdown system where you, you know, with mostly you you sort of use macros where you can just write, like, you know, potentially, like, you know, 3 sentences of of sort of custom content, you know, in a screenshot or something for, you know, if the product is simple to configure and obviously more if it's if it's more. And sort of all you need to do to write a new integration and have it have, like, a 100% test coverage. You can be very confident that that it's gonna work for all the different types of events that you need to do. And so I guess, like, sort of technically, like, a a lot of sort of our strategy has been how to take things that would become basically a bundle of infinite work that just keeps growing forever and, has a lot of duplicated code in it and and and ends up being a big source of pain for the project and trying to figure out how to make those things something where, you know, there's just not that much work that needs to go into them. You know, adding a new integration is a thing that, like, basically any engineer can read our documentation and do in a few hours, you know, at most and produce the same result that another engineer would produce, you know, that is correct as opposed to having different types of spaghetti codes. So I guess that's sort of you know, another sort of example of the surface area problem is that, you know, with mobile, like, building a separate app for Android and iOS is just incredibly

painful because every you know, even if you're just trying to make them exact copies in terms of the implementation, you know, so that you don't have to make as many decisions product management wise, You still like, you know, when you find a bug in 1, you need to, like, investigate whether that bugs in the other. You know, when you, you know, have a new feature, you know, in our in our, you know, Zulips markdown, which is the you know, we use markdown for our message formatting. You know, if we need to add it, we need to add it to both places and that sort of creates a bunch of release management headaches where, you know, sort of you don't wanna, like, create new syntax that's gonna make the Ash app crash, you know, and so you need to test on both iOS and Android to avoid that and and so on. So anyway. So moving to React Native has been huge on that front, but I, you know, I feel like we're we're sort of always coming up with with new techniques for how to sort of optimize

how much

of our time goes into actually improving Zulip versus into just the the work around having a large project.

And are you currently working full time on Zulip? Is it sort of your your full time engagement? And if so, how are you managing to support yourself, particularly given the fact that it's a fully open source project?

You know, I've been,

you know, had the good fortune of, you know, selling 2 startups to big companies, you know, you know, Caseplace to Oracle and and Zillow to Dropbox. And also my my,

you know, wife is a data science manager and, you know, makes a good salary. So I I don't have like sort of a super immediate, you know, desperate need to to make a salary. But, you know, I I do think that it's, you know, really important, you know, sort of to

make an open source project with sort of the scale scope and, you know, surface area of of ZULIP successful that you have some sort of, you know, sort of model for how you're going to be able to fund a core team that can work on that full time. You just want it to be the case that 1 can be confident that people will be responsive to important bug reports, you know, etcetera, etcetera. And, you know, I just I think like for a product of our scale, that's necessary. And the other thing is that, you know, a common piece of feedback that we hear is that, you know, people want to use ZULIP. They really like the sort of topic threading

model that we have, but they don't want to run their own server. So you know, or they wanna run their own server, but, like, you know, they don't really wanna be, like, on the hook if they end up with some problem that they don't know how debug and, you know, be sort of counting on best effort support. So, you know, I'm actually working on a a sort of company with the the sort of business model being to sort of solve those 2 joint problems. So, you know, we'll offer a a SaaS hosting service so that, you know, if you want to use, you know, Zulip, you know, you can have sort of the same type of experience you would have using a commercial product like Slack where you just go to a website and sign up and,

you know, you can pay some money to to in exchange for being able to use it. And then also providing sort of commercial support services for the the enterprises that are using Zulip. Because, you know,

if you're using, you know, something like Zulip for your group chat solution, like at a 1, 000 person company. Like, that that's a mission critical thing. It's it's about as important to people's daily productivity as your email server. And nobody would think of sort of not having a plan for what's gonna happen if, like, their email server, like, runs into some issue. You know? I mean, usually people just outsource it, but, you're gonna wanna have some sort of support contract. So that's sort of the the the long term plan there for sort of how to make this this work long term.

Yeah. 1 of the things that I found interesting and appealing as I was setting up a test instance of Zulip while I was doing research for the show is the fact that when you are creating a new organization on the server, it specifically asks whether it's for a company or an organization or whether it's intended to be community oriented so that you can determine the sort of gatekeeping that goes into play as far as somebody signing up, whether they need to be have an email that's of the same domain or if it's just anybody can self register. So as opposed to the Slack approach where if you wanted to be able to build a community

discussion with Slack, then you have to use some sort of trampolining service where you have a a website that will send you an invite after you enter your email where it's a bit of a workaround as opposed to having it built into the core of the platform.

Yeah. Yeah. And, of course, we we sort of offer those as sort of settings that are somewhat independent of that corporate versus, you know, community question. But, yeah, the main point of there is to be able to set, like, good defaults based on what sort of organization. And then also sort of from a policy perspective, you know, there's there's, you know, features that are really important for sort of corporate environments, like the ability for an IT administrator to read all of your messages, including your private messages that you might not want to have in a community organization. Like, a community organization, you know, very well might want it to be the case that, like, you know, people have some amount of privacy, you know, in their communications in that community. And so that that sort of it was primarily just about trying to set the right defaults and the right expectations basically for for how that community is gonna work.

So you don't end up with surprises where somebody thought that their communications were private, but actually, like, you know, the dude who set up the server, you know, has has access to everything that you've been

saying. And 1 of the things that you've touched on a few times is the fact that there is

is a very large and healthy community contributing to and using Zulip. So I'm wondering what are some of the methods that you that you and some of the other people in the project have used to foster and grow that, that engagement?

Yeah. So, you know, I think this is

a really important topic and and there's probably like 10 topics in this space that I could sort of talk about for for for, like, an hour. So, but but sort of at a high level, I I think, you know, if you're you're you're building an open source project and you want it to be like a a real healthy, you know, open source community,

you really need to optimize the project for the ease of contribution. So, you know, how efficiently does the project convert? Like a person who, you know, is technically skilled and is interested in contributing to somebody who has successfully contributed? And, you know, most open source projects do a really bad job of this. So I actually, been involved in organizing,

like the San Francisco Python

community sprints where, you know, we have like 50 people get together and work on various open source projects. And,

you know, it was amazing how, you know, some of the projects, you know, the you know, they were halfway through the day, you know, of a 2 day 1st day of a 2 day sprint. And, like, most the people didn't have their dev environment working in. So they hadn't, like, even attempted to start, you know, coding something. And I think this is just, you know, part of the reason that I organized these sprints is that, you know, it's, can come as a shock to the people who who run that open source project to see that happen where they're even there to help out and the people are still foundry halfway through the day. Because, you know, it really like,

you know, can help focus your attention on on sort of why it is that your project hasn't been attracting contributors. You know? Obviously, there's a lot of things that go into a project to attract contributors. You have to have people who, like, are interested in modifying the software, whether they be users or people who are just excited about the technology or something. Those people have to have some amount of programming skills unless you're you're gonna be teaching them programming. But, you know, there's just a ton of elements that go into it. So for for ZULIP, like, we've we've really optimized, like, long does it take to set up a development environment? You know, I don't know if you went through our instructions for that, but, you know, we actually like, you know, worked for the technical writer to make those instructions as good as possible. It's gonna cost a little bit of money. But, you know, I think it's 1 of the most important things that you can do as an open source project to improve your project is is making it the case that, like, getting set up as a developer is really fast. You know, it's also the case, and I'm sure you're familiar with this, if you're you're working on a, you know, code base, you know, you're gonna add into dependencies or add data mice migrations or other things where, like, sort of the environment needs to be reprovisioned

as you update the, you know, the environment. And we we have script that will do that reprovisioning, and we've we've optimized that script so that you can do a not provision, you know, if if nothing has actually changed in 3 seconds. So the the time being spent is only the time that it takes to actually install new dependencies if the dependency list actually changes. There's a whole bunch of sort of caching and and so on infrastructure that underlies this. You know, we we sort of, you know, put put a lot of effort in writing our code on readability.

So, you know, if you you sort of browse around the code of Zillow, you'll find that, like, we're very careful about how we name basically every variable in our code base. You know, very careful about how we name every function and very careful about avoiding, like, you know, sort of confusing duplicated things. Like, if there's 2 things that are very similar, they'll have a comment on top of both explaining like why we needed 2 of them rather than like 1 function that had a you know, argument or whatever. And I think like that that sort of can help make it accessible to people. And, you know, there's stuff about documentation. You know, we we,

you know, have invested a ton into tooling. Actually, I think tooling is probably 1 of the things that is is worth talking about. So, you know, most projects, you know, use some sort of linter to sort of make sure that people are using like a reasonably consistent style in terms of, you know, parentheses and, you know, or white space or whatever. But we've actually like built sort of a whole sort of mini linter framework where we'll run, like, the 8 different linters that we use in parallel. You might ask, like, why are there 8? Well, you know, you've got, like, a, you know, linter for our HTML for how the the white space formatting works. We've got a, you know, ES lint for our JavaScript. We've got, you know, the PEP8 linter for our Python code. You know, we've got a

a a puppet, parser,

linter to validate the syntax of our our puppet configuration, which is what we used for managing configuration and upgrading it and so on. And then we've also got, like,

you know, a few sort of custom linters that are things like a little framework where we can have, like, a 100 regular expressions

that we check that there's nothing that matches those. So, like, a common problem that you'll have in basically any project with new contributors is like they won't know that they need to tag strings for translation. And, you know, that's like a, you know, just a thing that like most people aren't taught about in school. It's really important, you know, if you wanna have a project like Zulip that's been translated into, like, a dozen languages, you know, you wanna make sure that when you add a new string, you know, that's user facing, that it is tagged for translation. But it it's just sort of like a huge pain to just, you know, with each new contributor have to teach them, that you need to tag your strings for translation and review. So we've we've just built a set of linters that will basically check, you know, if it's a string in a context that'll be user facing. Like, you know, it'll give them a little error and tell them they need to tag it for translation. And we've got, like, a 100 of those that we built up over the the last year. And that's actually, like, a huge part of what I spent my time doing when I only had 5 or 10 hours a week to work on Zoom nights and weekends was building out this tooling. Basically, anything that I was telling people on code review that, like, it was, like, clear that you could build a little tool that would, like, tell them to do that so that they would find that out without having to, you know, sort of spend both of my time and their time doing sort of review round trips on it. Like, we've we've sort of automated. And I think that's been really important to making

the case that, like, you know, some people will get started working on Zulep and, you know, get their first contribution in within, like, a few hours of when they start, like, downloading the the code base even to, like, including, like, debugging, you know, whatever bug it is. But, you know, I think that there's so, anyway, those are sort of, like, some of the technical elements.

You know, I also think that having a good community setup is really important. So we use Zillip, you know, obviously in the Zillip project and, you know, you can visit our cat. Zillip.org org community and there's

usually 50 something like that people online depending on the time of day. And it's really great because you can just ask a question,

you know, in development health and, you know, people will help you debug. And that debugging session is sort of it's your own space because you've got your own topic and and thread for that. And so that can go back and forth potentially for a month, you know, if you're working on a complicated feature. And, you know, it just is a very sort of convenient experience for,

both the the sort of contributor who has a place where they can send things and other people will apply, and they don't need to, like, worry about interfering with other people's stuff or or whatever or other people talking about things and trampling over their thing like you do have an IRC. And then from a maintainer perspective, it's really convenient because just, you know, I I wake up

in the morning and there's, there's a bunch of unread messages on different threads and, you know, I can just sort of reply to each of them and everybody's getting sort of generally quick, responsive, you know, personal attention from

experts on the project. And, you know, 1 thing that I probably should highlight on the community front is that I think responsiveness is really important. You know, a lot of people are really intimidated about, you know, getting involved and contributing with open source. You know, they like have it, you know, they really care about open source. They have a desire to get involved and contribute, but, you know, they spend their first day of their first contribution worrying about whether they've worded everything properly and, you know, just worrying about whether somebody's going to yell at them and so on. Just because it's, you know, it's really scary to be sort of putting your work out there and not knowing what's gonna happen. But then, like, a pretty common thing that happens after they get over all those barriers is just like nobody responds for a week. And, you know, my view is that if if people get, you know, a nice friendly response, like, well, you know, it depends on time of day, but, like, you know, ideally, you know, an hour later, you know, maybe, you know, 6 hours later, it just feels like a completely different experience as as a new contributor. And I think that that's really helped us retain, you know, so many talented people working on Zulip. Yeah. Having the guardrails available for when you first get started with a project, knowing that if you make a breaking change, you're automatically going to know about it and you don't sit there and wonder and worry about whether or not you're introducing a new bug to the system that might be a regression from somebody else's work that they've already done so that you can feel free to start

playing around and seeing what changes will have what kinds of effects. And then another thing that I noticed in your repository is the labels that you have for the issues, for where you're curating

issues for new contributors so that you have them available for somebody to get started with without having to have, the the entire context of the projects to be able to actually make a meaningful change.

Yeah. Yeah. Actually, the issue tracker is is probably 1 of the more important things of of my strategy on the front of making Zillow friendly to contributors. So, you you notice the thing about sort of bite size you know, we tag in issues that are sort of bite sized and accessible.

A few other things that we do that I I highly recommend.

1 is that we have this, you know, GitHub workflow bot, which we call Zalub bot, that basically works around a bunch of that sort of permissions and notifications issues that GitHub has. So, like, as an example, like, in GitHub, you can't give somebody the ability to assign an issue to themselves unless you also give them full write access to the repository.

So, basically, like, you you can't just give some new contributor who, you know, just showed up on your project full write access to your repository because that just feels a little bit crazy. And so effectively, like, there's no way for a new contributor to sub and claim an issue. So anywhere I bought, you just post a comment that says at Zulip bought claim on the issue and it'll basically add you as a read only collaborator to the repository.

And And then it turns out you can assign issues to read only collaborators even though they can't assign them to themselves. And, anyway, there's a bunch of sort of things like that that that we do with Zoloft. It's actually like a totally configurable framework. So if anybody's interested, you can, you know, check it out and use it in your project. But, you know, basically trying to sort of cut down these barriers to feeling like you can be the maintainer for a subsystem. Another sort of example of that is we have a system of, labels for the different areas of the project. You know? So, you know, if you're sort of feeling overwhelmed by everything going on Zulub, you can look at, okay, like, you know, what are you know, look at the markdown issues. So you, you know, maybe you pick 1 bite size issue, and then you can sort of continue and just work through all of the issues in our markdown projects as you sort of gain experience in that subsystem and, you know, eventually become the maintainer of that. And, you know, each of these sort of areas we try to keep under, like, 30 open issues or or something like that, and we'll split it if it gets too big. And that's been a huge help in in sort of getting people onto these roles of sort of subsystem maintainers,

is just having a way for them to subscribe.

So 1 of the things we do at Zillow Bot is is, whenever an issue is labeled with 1 of those labels, it'll will send a notification mentioning a team. Anyway, it's it's basically a hack. But, basically, it's possible to subscribe to an area, labeled within Zulot, on GitHub, which is not possible with GitHub,

in general. But the final thing I wanted to talk about with, you know, with with issues is

a lot of projects will the things that they put in their issue tracker will be, like, you know, a 1 sentence description of a bug or, you know, maybe, like, you know, a screenshot showing a a bad behavior or something like that. And that'll be sort of the issue that a new contributor encounters when they're looking at whether they wanna work on it. And, like, I don't know. If you look at, like, a screenshot showing some bad behavior, like, you have no idea, like, what's gonna be involved in doing that. And so you're gonna, like, look at that issue and be like, I don't know how to do that and bounce through, like, 75 issues, you know, before you find something that you feel like, oh, I can I can get started on this? I can work on this. So a big part of, like, what I do is whenever a new issue is opened, I'll I'll post a, you know, paragraph long comment on it that explains, like, okay. Like, here's, you know, the section of the code that this is involved. Like, I'll often do a pointer to 1 of our documentation articles. We have, like, a 100000 words of developer documentation, which, you know, I think is is really valuable. And I'll talk a little bit about how we we sort of put that together together later. But basically put together sort of the the data that you would want as a you know, if I don't know. If you had a new hire at your company and you wanted to give them their first day project, like, what is the stuff you need to tell them to get started? Like, okay. Here's the doc that explains how the system works. Here's, like, the file that this issue is probably in, and here's, like,

some high level thoughts about, like, roughly how I think we should fix this because, you know, a lot of things, there's, like, different ways you could change the behavior, and it's, like, not really clear. But the overall goal of that is to make it the case that, like, somebody can you know, I'm looking at an issue if you're, like, a reasonably skilled programmer and the languages and stuff involved. Like like, you could work on this issue. Like, you're not. It's not just sort of a thing that's impossible. So we have, like, 1 of our pieces of documentation is a new application feature tutorial that, like, explains how to add a new setting checkbox in Zillip and, you know, sort of goes through the whole flow of of how our real time sync process works and all of that. So that, like, it both sort of teaches you how that works, but it's also like whenever there's a new setting, you know, I'll just link to that document. Then, you know, they they only need to know how to implement the behavior that that setting is supposed to control. And the rest of it is sort of, you just read the doc and you can you can figure it out. And what have been some of the most unexpected aspects of working on Zulip, whether it's from a technical perspective of things that you've learned or from engagement with the community

and just learning things from the people that you're working with?

Sure. I guess 1 unexpected

technical thing is, just how complex a correct,

implementation of emoji,

you know, for in a group chat application is. There's just, you know, just amazing number of of different details. You know, you you have both, you know, the emoji that you put in messages and emoji reactions, and you want those to share a sprite sheet. So you need to figure out how to do the, like, scaling, you know, correctly.

You know, And we ended up with some scrolling performance issues because we used the wrong CSS for scaling, so we needed to fix that. And then there's sort of the like, how do you name the objects? Because you know, emoji are are sort of nice. They all have sort of a Unicode series of code points that defines, like, what emoji character that is. But, you know, we we actually support, you know, uploading custom emojis.

So then, like, where do you put, like, the ID for that custom emoji and that sort of code point structure? And, you know, it's just there's just an amazing amount of complexity that is sort of hidden behind this. And that that doesn't even get into the issues of, like, how do you type ahead for for a sort of which emoji that you're trying to select when, like, you might be searching for the thumbs up emoji, but maybe you think it's called plus 1 or maybe you think it's called thumbs up. And, like, you know, how do you sort of avoid having duplicate copies of it in your emoji picker? Because that's sort of silly and and looks bad while while still sort of supporting people using either of those 2 names. And anyway, it's just, it's it's been sort of fascinating, like, just how much total,

you know, sort of code, you know, ends up going into that, you know, in in every single project. You know, sort of on the on the community side, I think 1 thing I've been surprised by is you see a lot of sort of bad behavior in some open source projects. You know, I think both, you know, the Linux kernel

community and the, you know, deviant community have had issues at various times in the past with, you know, being unwelcoming to to new people, you know, you know, some people being a little bit of bullies or or whatever.

And,

you know, it's sort of I guess I'm sort of surprised that we've made it to 300 contributors and there's basically you know, we have a code of conduct, but I think we haven't had a single code of conduct indication about some particular thing that somebody was doing in in, you know, a year and a half, you know, with with hundreds of people. And, I don't really have a good explanation for that. I mean, I think part of it is that I I think we just produce such a sort of friendly and welcoming vibe in terms of how people communicate on ZULIP that maybe we haven't attracted troublemakers yet. But, I don't know. That's sort of been interesting for me because it is a, you know, a thing that I've definitely seen in communities in part of life. I've been involved in both the Longxternal and Debian.

But, you know, I'm just very glad that it it you know, we haven't had to deal with that ourselves

yet, I guess. And what are some of the things that you've got planned for the future of Zulip as you continue to develop and grow the community? Sure. The technical base as well? Yeah. I mean, you know, I I think I said at the beginning that, you know, Zulip is the the leading open source,

modern group chat tool. And, you know, we really wanna delete open source from that

story. You know, I think my experience is that people who've used both Zulip and something like Slack, you know, they don't want to go back to using Slack after they've experienced the the sort of efficiency that you get from having topics in your your chat. You know, it's sort of like the, you know, experience that people my age at least remember when you you first got to use, you know, a threaded email client like Gmail after having tried the other webmail stuff. Right? This is it just feels totally different. And, you know, I think that there's,

you know, a long way to go on that front in terms of marketing and and also just in terms of, you know, just the total engineering required to have, you know, that many billions of people using your thing, I'm I'm sure that there'll be many more, you know, sort of issues that are currently just you know, haven't been important because we don't have, you know, as many people in funny use cases using Zulu, but, you know, as your user base grows, you get more and more requirements. But I think that that's 1 part of our vision. You know what? I think the other thing that's important to me is,

really helping support

the the idea that, you know, if you're building an awesome new technology in 2017,

you know, you should be building it as open source software because that's

sort of the right thing to do. So you know, I think,

you know, part of my sort of vision is that if if ZULIP is successful in becoming the the leading group chat solution worldwide, like, I think that that

or I hope that that would help,

you know, inspire people to, sort of, you know, if even sort of, you know, a a unicorn with $500, 000, 000 you know, can be beaten by the power

of a great open source community, you know, I think that that would have an impact on people's perspective on things. So I guess that's sort of

part of my sort of personal goal. In terms of sort of the company, you know, I think it's important to build sort of a self sustaining

profitable company

that gives us the sort of flexibility

to maintain Zillow as completely open source software. You know, I've talked to, you know, a lot of companies that are sort of built around an open source projects that have taken VC funding. And

I generally

it seems like they have a very hard time, you know, in practice, you know, trying to actually really stay focused on the open source project. You know, often it'll become basically impossible to contribute to somebody who's not part of that company, you know, sort of like how a lot of Google's open source projects work. So I don't know. I guess those are some thoughts on on where we're hoping to go. Alright. Are there any other topics or questions that you think we should cover before we start to close out the show? Oh, I guess we can come back to the the sort of documentation thing we mentioned earlier. So, so ZULIP has like a 100000 words of of

100000 words of developer documentation.

And, you know, we had basically none when Zulip was open sourced 18 months ago. So this has mostly been written in that time period. And, I guess sort of like my philosophy on this is,

you know, whenever somebody

wants to work on some subsystem that is complicated enough that it needs to be documented, but like not like, you know, isn't, I guess. Like, the the sort of correct answer as the expert is is not to, you know, sort of spend an hour with that person, like, explaining it to them. Like, the correct answer is to spend an hour writing some documentation on that and then send it to them. And I think, like, with with that strategy,

you know, as sort of the primary strategy, you achieve sort of 2 goals. 1 is that eventually you end up with pretty good documentation on on all the things that people are working on in your project, which is sort of what's important. But the second thing, which is a little bit more subtle, you know, for but it's important for an open source project is that you don't end up wasting time as a maintainer. So like 1 of the things that,

you know, you find if you don't do that is you have some excited new person shows up who wants to build feature x. You know, you spend several hours of your time teaching them enough about, you know, how the rest of the system works so that they can build feature x, but then they end up actually too busy or not skilled enough or not interested enough or whatever to actually build the feature. So you sort of wasted 3 hours teaching this person who didn't end up, you know, actually doing the thing that they

they sort of were intending to do. But if you instead, you know, document it or or or, you know, explain, you know, sort of your ideas on the issue tracker or something like that, you know, that's fine. You know, if that person ends up not having time for it, you know, maybe somebody else will come by later and they can benefit from the work that you did. So, anyway, I guess that's sort of,

another tip that I I think should be very helpful to a lot of people. I actually use that, also at CaseBlaze when, you know, as sort of part of the preparation for for my departing, you know, you know, I'd sort of been running the technology and, you know, I really didn't want it to be the case that, you know, people weren't able to to sort of go on with full understanding of how the system works. So, you know, I just spent 3 months where whenever anybody asked me a question, I only responded in the form of documentation,

not not in the form of answering their question. And that's worked out pretty well too. Yeah. I listened to an interview a little while ago with someone who was saying something along those same lines where anytime somebody emailed them with a question, they would never just respond

via email to the question. What they would do instead is write a blog post of the answer and then send an email with a link to that post so that other people would then be able to take advantage of that same knowledge without them having to duplicate the effort or go hunting back through their email to find where they had answered it

before. Yeah. Yeah, exactly. And I think a key part of my process, I guess, is that you ask the person to do sort of like, basically to suggest edits to the documentation

after reading it. And I think that's really important too because, you know, I don't know. I guess my viewpoint is that, like, anything that you produce that has not sort of been verified in some way is probably really bad.

And so, you know, it's just really great to take advantage of the fact that you have a person who actually, like, needs that documentation to get something done, to to get, like, like, a review that isn't just a cursory reading, but it's like they they're actually trying to understand the system, and they they they wanna know the answer. And, you know, that that principle is sort of the same principle as to why I think it's really important if you're an open source project to do sprints or if you're a company to like have the people who work on the developer join actually watch new people try to get involved with the project, like, sitting over their shoulder is like you know, that's the way that you test the experience of getting started is working on your technology. You know, you have to actually observe somebody doing it. So,

unless you've got anything else to add, then I'll start to move us to the end of the show. Sure. So for anybody who wants to get in touch with you or follow what you were up to, I'll have you add your preferred contact information to the show notes and a link to the,

meta discussion site for anybody who has questions about getting involved with the project. And so I'll move us to the picks. And my pick this week is the LEGO Mindstorms

EB 3 robotics kit. I recently picked it up for my son so that we could start playing around with that and learning some robotics and teaching him some programming. And so far, it's been very fun and entertaining, and there's

a lot of really interesting things that you can do with it. And the logic brick itself is actually hackable so that you can load in an alternate an alternate firmware to be able to run Python on it. And there's actually somebody who created a interface

extension board for Raspberry Pi, so you can actually just hook a Raspberry Pi into the Lego components themselves and power all the servos and sensors that way. So it's definitely a pretty interesting set of materials and, capabilities that you can build off of that. So with that, I'll pass to you. Do you have any picks for us this week, Tim? Sure. Yeah. I have a couple. So, 1 is a book,

which is The Checklist

Manifesto by Atul Gawande. It's I guess, like, sort of my

perspective on, you know, I think, like, if you're if you're trying to to build a really high quality process, you know, I feel like your your sort of first thing that you attempt should be to to automate things. You know, make it the case that it's it's not possible for a human to screw up by forgetting to do something or doing something wrong or whatever. But, you know, wherever that's that's not possible or if you have a thing whether, you know, it's it's very difficult to do. Then the next best thing is a checklist. And, you you know, throughout my career, I've found that, you know, if you really want to make something high quality and there's any sort of human series of steps that needs to happen there, you know, you should use a checklist. And it's just a great book that sort of, you know, it's very short and sort of teaches you both, you know, how

to be awesome at checklist

and maybe help you explain sort of how valuable they are across a sort of wide spectrum of industry from, you know, medicine to construction to to to everything else. And, you know, the second is, you know, about cooking. So, you know, my wife and I cook at home a lot. And, 1 of the things that, you know, you find that, you know, if you cook a lot is that, you know, often you'll have some recipes that you find, that are pretty great, but you end up modifying them based on your preferences when you actually make them. And it was really good, but you, like, want to be able to make it again later. And,

you know, often people end up with, like, you know, this problem that, you know, you found recipes on 20 different sites or in books or whatever.

And our our solution to that is that we have a Wiki basically just for my wife and I to keep track of our recipes. It's at, recipes.timandallia.com.

And, you know, we just find that it it's fantastic because, you know, we can keep track of exactly, you know, what it was that we did so that results are reproducible, you know, and, you know, it's just super valuable for debugging and producing, you know, sort of what you want. And it's also super easy to share. And,

it's like very little effort. You know, usually we'll start out, you know, with something. We'll just put the link to the source recipe, and then, you know, you start adding notes on on what you modified and how well it worked out and so on. And, I don't know. I I think it's, you know, if you're into cooking, it's a a pretty good technique that,

you know, can make your life better. So Alright. Yeah. It's definitely an interesting approach. So great. Well, I appreciate you taking the time out of your day to share the story of Zulip

and the technology behind it and how you're growing the community. It's definitely an interesting project, and I've actually installed

my own instance of it. So for anybody who wants to check that out, I'll leave a link to that in the show notes so that you can join the podcast and the chat community there. And I, yeah, appreciate your time, and I hope you enjoy the rest of your day. Yeah. Cool. You know, thanks so much. And and, also, if, for anybody interested in checking out Zulep, you know, you can just stop by our community at chat. Zulep.org

and sort of see it in action with our a large active community there, which maybe your community will be soon. But, you know, for now, it's it's just a a really nice way to sort of see the the whole thing in action.

The Python Podcast.init

Summary

Preface

Interview

Keep In Touch

Picks

Links

The Python Podcast.__init__

Summary

Preface

Interview

Keep In Touch

Picks

Links

The Python Podcast.init