Summary
As our system architectures and the Internet of Things continue to push us towards distributed logic we need a way to route the traffic between those various components. Crossbar.io is the original implementation of the Web Application Messaging Protocol (WAMP) which combines Remote Procedure Calls (RPC) with Publish/Subscribe (PubSub) communication patterns into a single communication layer. In this episode Tobias Oberstein describes the use cases and design patterns that become possible when you have event-based RPC in a high-throughput and low-latency system.
Preface
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable.
- When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for running your awesome app.
- Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.
- To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
- Your host as usual is Tobias Macey and today I’m interviewing Tobias Oberstein and Alexander Gödde about Crossbar.io, a high throughput asynchronous router for the WAMP protocol
Interview
- Introductions
- How did you get introduced to Python?
- What is Crossbar and what is the problem that you were trying to solve when you created it?
- What is the status of the IETF WAMP protocol proposal?
- Why have an open protocol – and how do you see the ecosystem?
- Python isn’t typically considered to be a high-performance language so what led you to use it for building Crossbar?
- How is Crossbar architected for proxying requests from a highly distributed set of clients with low latency and high throughput?
- How do you handle authorization between the various clients of the router so that potentially sensitive messages don’t get published to the wrong component?
- Does Crossbar encapsulate any business logic or is that all pushed to the edges of the system?
- What are some of the typical kinds of applications that Crossbar is designed for?
- What are some common design paradigms that would be better suited for a WAMP implementation?
- What are some of the most interesting or surprising uses of Crossbar that you have seen?
- What do you have planned for the future of Crossbar?
Keep In Touch
- Mailing Lists
- #autobahn on IRC
Picks
- Tobias
- Alex
- Tobias
Links
The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA
[00:00:14]
Unknown:
Hello, and welcome to podcast.init, the podcast about Python and the people who make it great. I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy, so you should check out linode@linode.com/podcastinnit and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or trying out something that you hear about on the show. You can visit the site at www.podcasttinnit.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. And to help other people find the show, you can leave a review on iTunes or Google Play Music, and tell your friends and coworkers and share it on social media. The host, as usual, is Tobias Macy. And today, I'm interviewing Tobias Oberstein and Alexander Gode about crossbar dot io, a high throughput asynchronous router for the WAMP protocol. So, Tobias, could you please introduce your I guess I should probably redo that and pronounce your name properly. So used to people anglicizing my name. So
[00:01:04] Unknown:
That that's awesome. I'm that's the first time I've been interviewed by by another person who's, with the same 4 names. So my my name is Tobias, and, I'm cofounder of Crossbar. Io, of both of the project. And, recently end of year last year, we founded a company focused on that product or the project. And the co founder is Alexander Goede as, as he mentioned it, hopefully he will be able to join in. And, Yeah. Well, what should I say? I'm into computers since a long time. So my, my originally introduced, was, by my stepfather who brought, some single board computer to our home. That was, I was like 12 years and then got into hacking.
And, well, then I've professionally, I've been doing a lot of consulting work for big companies in in data warehousing and other stuff. But in the, in the last couple of years, I've been much into Python. And then I've, we've started, well, probably another project besides crossbar, which is known to, to some guys probably that, which is Autobahn, which is, well, it's, it's a, it's a collection of web socket libraries and, Womp libraries. And yeah. Well, that's probably enough to for for intro intro to my person. So And how did you first get introduced to Python? I was hoping you wouldn't ask me because honestly, I forgot. I was trying to, to, reconstruct when the year when I've started to use Python, something thing around 2, 002 because at that point I was still in a master theaters and, that was mainly C plus plus but I, I wanted to wrap that making it, easier for, for other researchers to approach. And that was the point.
I definitely know I was using Python at that point. And then I was using, other languages over the years. So I left Python for a couple of years and then came back. So now I'm pretty much using it every day and for the last couple of years. Well, I I'm, I not, I not left C plus plus completely, but, every time I'm coming back, it's like, you know, you get used to the simp, simplicity of Python and the it just works. So it's doesn't stand in the way. So I'm a I'm a heavy Python user
[00:03:35] Unknown:
since a couple of years now. So, yeah, if you could just give us a brief introduction to what is Crossbar and what was the problem that you're trying to solve when you first created it. Crossbar nowadays is positioned as a multi protocol application
[00:03:49] Unknown:
router. So what's that? So the idea is, you have a distributed application consisting of components, or you have, application built around the microservice microservices approach. Approach, then then you have a need for how do those components services communicate. And, crossbar provides, infrastructure for the communication between application components and, and microservices. So, that's the application router part and the multi protocol part is, as, as you mentioned, it's, it's based originally based around a WAMP, which is based internal web socket.
And that protocol was also started by us. It provides 2 application messaging patterns. So remote procedure calls and publish subscribe and, Yeah, well, so crossbar aims to provide a high level, communication infrastructure for apps and has the aim for seamless connectivity. So that's, that's our goal. Seamless connectivity in our view means crossing languages. So it's not Python only and crossing network and system boundaries. So that's, that's the, the scope and the original problem we had was we, we wanted it that's years ago, some years ago we wanted to build a mobile app and, and the app should talk, should have talked to your back end end and at the same time have a browser front end talk to that same back end. So then you have a multi language scenario or we had that multi language scenario and we were, we've been looking for a protocol to make that happen, to connect those front ends to the backend. And not only do that, but have a real time, ability for the user interface. So being able to update the user interfaces in real time from the backend. And at that point, that was like 2011 or 12.
We've been looking around for suitable protocol and didn't found anything or basically was all hacks. So the, the, the web part or a browser front ends, there was combat and other hacks to get real time capabilities. But at that point, WebSocket started or the, the ITF started a WebSocket for a working group to, to come up with a new protocol for the web, which is bidirectional real time. And that is WebSocket. That was the point where we started to where, well, crossbar started to develop. So, and nowadays we that old project, that old app it's long quit. So we're not we're not, developing that anymore. But now the focus is just on that problem, the communication problem.
[00:06:36] Unknown:
And I noticed when I was going through the documentation to research the show, I noticed that you had started an actual IETF draft protocol proposal for the WAMP protocol. So I'm wondering what the status of that proposal is and if you ever made it over through to having be an RFC. Yeah. That's, it's a very good question
[00:06:55] Unknown:
because well, the status is like this. The working group for WebSocket, has fulfilled the original goal, the original chapter. I think they call it chapter or something. And, so the chair of the working group, not close to working group because it reached that goal. And so we are looking for new well, to develop an RFC in the I t f you need a working group because you need exchange between different implementers. And, we've lost, that is kind of not stalled, but in the, in the waiting queue for us, it's we have so much to do. So we we want to to, to take up the work again in within the ITF, in that working group to make it into a proper RC, but that's still a lot of work.
So currently we, we just don't have the time to, to push that forward, but it's on the we definitely want to because, I I think it would we we already have a lot of, alternative implementations, which is you normally the the the biggest hurdle to get into to get an RSC because the ITF has that policy of you need 3 independent implementations. And so that's normally the biggest hurdle or barrier, but we already have that. So it's, it's a lot of editing and publishing work. I've been in the work in the ITF working group for web socket was like 1 and a half years intensive discussions. And it's, it's really a lot of work. You can, can't overestimate that if you, because you get into exchange with very, very smart people.
And obviously they have, hard questions and hard critics and going through this whole process is a lot of work. So we want to do it, but we don't have the time right now. So if there's anyone who's, saying, oh, I'm into network protocols, probably you want to have, have a chance to be, like coauthor or co editor of such an IRC. That will be awesome. So if there's anybody else out there, just get in touch with us. So
[00:09:12] Unknown:
What prompted you when you were first drafting
[00:09:23] Unknown:
Oh, that Oh, that that's a very good question. I I I think it's, me, I was like me personally, I wouldn't use anything which isn't an open standard or which tries to lock me into something. As a as a developer, I was like, let's we there isn't any, any point in making it closed or developing a closed protocol, I think nowadays. So it, it needs to have, alternative implementations. And to, to do that, you have to have an open protocol. So we we said in the beginning, and we also said, but, we don't want to lock developers into our stuff. So there should be different alternative implementations for Womp. Not only so if you say I want, I like WAMP, but I don't like the crossbar or Autobahn stuff. That's cool. You can you can use a non cross bio WAMP router and you can use non auto bond client libraries. And, we totally support that.
Of course, we say our implementations are great and and and we're good, but, you you shouldn't be locked into. So, I'm pretty much convinced, even if I was talking, like some guy once asked me, why are you like like marketing, alternative implementations? Doesn't that hurt you, sir, your own company? And, I'm pretty much convinced. No, it doesn't. It it's on the law in the long run, it's necessary. So you should be able to have, not being locked in. So that was the idea from from the very beginning. And then it's a thriving ecosystem. So there are new client libraries and and also router, implementations coming out, all the time. So just in the last days, it was, another C plus plus client implementation announced. So yeah. Alexis.
So, people all like the the the, days before or weeks before was like Pearl was another we now have a Pearl client library implementation for WAMP, which is awesome. Awesome. I'm I'm not using WAMP, or Pearl obviously, but, others are doing and then, and those are, so the gaps are closing. I I think it's a healthier ecosystem. It's, it's growing. And, well, may maybe that's also an important point. We have a dedicated webpage where we, where we really, well show have, have a list of implementations and we don't have a policy whatsoever. If you say you, you, you have a WAMP implementation regardless client or router, and you're aiming for being compatible, then we will put your stuff, link your stuff on the WAMP website.
So there's no discrimination in any way. The only thing we say, if, if you're trying to modify the protocol, that's also cool, but then don't call it 1 because that would confuse probably confuse users then. So we we reserve, we still reserve a last word in, like like, you using the womp trademark or that word, just to avoid confusing users. But you could even frog the protocol if you wanted to, which didn't happen today or out of up to up today, but, you could. And yeah, well, so I, I think, and there's also mailing list, which is, has quite some healthy activities. So yeah, I would say language wise gaps are now pretty much closed. Of course they're, like esoteric or I think we even have a Haskell implementation or something.
So there's still some gaps left, but they are closing. So
[00:13:04] Unknown:
That's That's how you know you've made is when somebody takes the time to actually implement your protocol on Haskell.
[00:13:10] Unknown:
Yes. We have have it. I I see it has to go, so it's done already. So, but, Yeah. And and to your point earlier as well about
[00:13:21] Unknown:
having the multiple different implementations and competing implementations as well, the rising tide lifts all boats, and it's a similar situation to the conversation that I had with the folks at Sentry where they have a hosted offering, but they also offer the exact same product as an open source self hosted site. And because of that, I think is what leads to a lot of their actual usage Because as you said, there's no lock in. People can start using 1 implementation and then easily shift to another 1 without having to completely rearchitect their entire application. Definitely. Definitely. So for example, with the Audemann libraries, we try to bring over the developer
[00:14:01] Unknown:
experience over different languages or as close together as possible. So that bit, but that's only 1 like which leads to certain style in, in the Autobahn C plus plus for example. And, but there definitely is room for different approaches. So we they're competing c plus plus implementations, which are optimized in a different way. For example, qt or not using boost. So the this last library in c plus plus they had the issue like, c plus plus boost is a, is a monster. So you don't want to, it's cool, but you don't want to have it always. So there's room for, for different, even in the same language. So there, even if, if you take just C plus plus or JavaScript, there's also alternative JavaScript, 1 implementation.
So I'm, I'm a totally total fan of that because, well, you have choice. Developers should have choice. And, if, if there's another 1, the other guy coming around with a better JavaScript experience than Autobahn. Js. So that's cool. Why not? So we open to that. That's that's I think that's cool.
[00:15:10] Unknown:
So the reference implementation and crossbar was written in Python. So I'm curious that given the goals of the WAMP protocol is to be fairly high throughput, I'm wondering why you ended up choosing Python as the,
[00:15:23] Unknown:
language of choice for the reference implementation for something that is intended to be high throughput and low latency, and also how you manage to achieve those goals while using Python? Yeah. That's a very nice question because I love it because I love like talking to the Java guys, Java people. I'm still using Java, but they're always like, oh, we have that cool JVM hotspot. Nothing is like that. And stuff like this is all, this is old. This is like the world is it's not the same, like a a couple of years ago. And people probably haven't, or not all developers have like acknowledged that tiny scripting languages like Java script or Python have come a long way also performance wise. So I'm, I'm coming from the C plus plus side of things. And so I I'd I'd say, okay, I'm I I know how to do bare metal stuff, but, Python offers just such a a bigger or better developer productivity compared to like in particular C plus plus so that's a main point for us productivity.
For example, crossbar, if you take everything, Autobahn, it's like 50 ks lines of code And it's for that. It's doing a lot. And if we had it implemented in C plus plus it would be multiple of that. And we we couldn't have pulled that off as a small company or with a handful of of developers. So the productivity is, is much better. Like performance wise, if you, you can, like in, in, in Python, you have different implementations. So there's C Python, which is the classic interpreter. But nowadays we have PyPy, which is a tracing chip compiler. And this is what we use in production.
So it's the crossbar and Autobahn in production. We usually run it on PyPI. So it's much faster than C Python because it's, it's, compiling to native code in like Java, the JavaScript engines and the browsers. And, that's a major point performance wise. And then we have, like native tiny bits of native code interface. Like for example, in the, in the web socket implementation, you, you have to have a masking and that's, you can accelerate that still accelerate over getting it faster than the JIT can do, but that requires like vectorized code and stuff like this. So it's, we don't have, I'm pretty sure I would take bets if, if someone is, is like, wants to beat crossbar its own game. I'm pretty sure it will be competitive. So yeah. Well, what in that area, what else could I say? The the commercial maybe that's interesting. The commercial offering of crossbar has, has native code acceleration, which which is, yeah. Well, adding, another level of performance. But if you if you'd ask me if, now in in looking back, would we rewrite crossbar in like another language?
And, I was asked at, at 1 point by some guy and the answer is I would only consider 1 other language and that's rust. So, but that's not on the on our road map. So and we don't have a need right now. So it's it's fast. What we want to what we need to add is multi core, multi node support, but that's not a question of implementation language. It's more a question of getting all the bits talking together. And so no. We don't have a problem and I would take any challenge in beating cross by its own game, like from the Java guys.
[00:19:10] Unknown:
So could you share a bit about how Crossfire is architected and some of the underlying technologies that you're using for allowing it to manage that high throughput and also the, sort of highly distributed nature of the client traffic as well?
[00:19:24] Unknown:
So that's actually 2 questions. So internally, Crossbar has a multi process architecture. So that's, a crossbar node always starts with a controller process and then forks off worker processes. So we have the ability to scale on multi core by using multiple processes. So that's, that's a major point. And an important aspect in this is that we also using Womp internally. So the, the different worker processes in a crossbar node, they are talking to the node controller process also using WAMP. That's probably that recursive nature. Alex has recently told me that it's took him quite some, some time to get behind that story, but it's that's we using wamp internally to in the process communication for the in the process communication. We're also using Womp for the upcoming, management, cluster manager. And, that's also important part maybe that, or that touches that second part of the of your question, highly distributed sets of clients. How we how do we treat that? That's really our vision. We we think it'll, applications in that area of the Internet of Things or, industry 4.0, it's called here in Germany, is you have inherently distributed application with you have different locations, factory floors, you have a cloud back end ERP systems, you have mobile devices and all that. And our reason is not to connect all those end points to 1 crossbar node running in the cloud, but to have also a network of crossbar nodes also distributed.
And that answers or that touches that low latency questions. Because for example, imagine you have a machine on a, sir, on a certain factory floor in the location a, probably that machine is wrapped as a warm component and should be able to talk to a backend on the local location on that plant. But it should also be able to talk to a cloud back end. So you have a low latency for the local communication to the local backend running on that location a. And you have a necessarily bigger latency if that wants to communicate. The shop flow wants to communicate with the cloud backend in, in, in the data center. So it's not a question And you want to have both, you want to have the ability to, to let it talk to your cloud, but also have low latency, high throughput locally. So you want to have a it's it's called like in in there's a term like edge cloud, FOC computing.
So the idea is to extend the cloud right into edge locations. I have to say we're we're still not there. So we that's, I'm I'm talking now about the vision where we want to go with crossbar. It's not ready today. What we are doing today for in the latency area is, like, for example, for our own demo instances, we're doing latency based DNS. So if a client comes in and wants to resolve the host name for crossbar, it's directed. The DNS is resolved based on the latency to the crossbar node. So if you're in the US and the opening or demo page, you should be directed to an US instance where you get lower latencies, compared to as if you were connected to the European instance. So we have multiple instances around, the globe and you're directed to the at, already at the DNS level to the nearest instance.
So that reduces latency because like going from here, Germany to west coast us is like 250 milliseconds, round trip, or 200 to 215. Possibly that's just physics. It takes it's like a couple of 1, 000 kilometers. So you can't, trick, speed of light. So that's basically, and that works today. So you can, we using route 53 AWS and it has a cool feature of latency based DNS. You can do that today, but, the other story is still the bigger 1 that distributed network of routers, and we are not there yet. That's a vision, and we're working on that.
[00:23:49] Unknown:
Yeah. There are a lot of complicating factors for that, not least of which is service discovery for being able to figure out where the destination point is for the remote procedure call, for instance, and figuring out what network of routers you need to transmit the message through to be able to reach that destination point. So
[00:24:07] Unknown:
But but that's that's just an very interesting aspect you're touching on the, on here because okay. Yeah. That's absolutely right. We, we have to crack that nut and that's part of the problem, But the whole issue of service discovery, like, if you're in the in the Http rest world, if you're doing microservices using the rest patterns, then all your microservices are servers, our web servers. And then you have the problem. If you want to call that, how do you discover it? And the whole, category of, of, of apps is, or is, is targeting that problem. It's called API gateways. Like, NGINX has, has some product in, in there, but, other other companies also like works like this, the, the service, the HTTP rest server registers itself at that gateway API gateway, and the gateway remembers where's the implementer of, of that servers. And then if your HTTP rest request comes in, that is, rooted to using that information to the right web server endpoint. And we don't have that whole issue in crossbar because WAMP has not only a RPC, but what we call rooted RPC.
So the the colleague side already tells the 1 prooter or the 1 prooter already knows where the, the colleague resides at the point of problem disappears with Womp. You don't have the service discovery problem. You the caller just calls, 1p u, URI or the procedure is identified by, by 1p URI and the 1p router already knows where how to forward the call. So this is and it's also the biggest difference with we we had WAMP has has had in history 1 major upgrade from 1 version 1 to 2 and okay. We don't expect the version 3, so that's, but it was a big jump. And the, and the main point in there was we've moved from RPC to rooted RPC.
So originally 1 1 was like, you had to implement your procedures as part of the router. It was, was like very inflexible and that was the biggest change. We split that out. And the nice side effect is we don't have the whole issue that you have in HTTP rest with service discovery. That's a nonissue
[00:26:34] Unknown:
for us. Yeah. And it also seems that that would help prevent the issue of the right message ending up at the wrong destination so you don't have to worry about potentially leaking sensitive contents of a message because you know exactly where the message is going to end up at the destination point without having to worry about it being broadcast and then Yeah. Relying on the endpoints to filter earlier. Yeah. Yeah. Yeah. No, that's exactly the case. So, from the router point of view,
[00:27:02] Unknown:
client components are untrusted in the sense they have to authenticate. And then, the router knows, has assigned an a role. So we have role based authorization in crossbar and then you have the ability to to configure the permissions in the router and the router enforces those permissions. So you can say on the 1 URI, the 1 role is allowed to subscribe, but not to publish. And then he even can have a publisher only allowed to be able to publish, but not to subscribe to its own events. So and the subscriber side, you can you can make it only a subscriber and disallow any publication. So that allows a quite fine grained, control over the internal also the internals of the application. So you can, like, like, enforce rules internally, not only to the outside world. So
[00:27:56] Unknown:
My understanding is that at least in the initial implementation, crossbar was based somewhat heavily on Twisted for handling asynchronous requests. And I'm wondering, 1, how you're leveraging Twisted, and then, 2, what your view of the asynchronous landscape in Python great question. It's okay. They have multiple aspects. Okay.
[00:28:20] Unknown:
To a cross bar itself is, is using twisted and it's using, in particular, it's using Autobahn in, in its twisted flavor. So Autobahn Autobahn Python supports both twisted and asyncio as the underlying network framework. So you can you can choose. But in crossbar, we've chosen twisted and, and that's, there are multiple reasons. 1 reason is we we wanted and still want to support run, running crossbond on Python 2.7. And then the more important point, Twisted is much broader in scope than asyncio also. It, it brings a lot of protocols with it. It brings a web framework with it. And, so we using that. And for example, it has first class process support, which is trick kind of tricky with if you take in signals and, and stuff like this, getting that right, isn't easy. And it's it's, they've done so much work over so many years, the twisted guys. It's it's great. It's totally underrated. It's probably, the documentation isn't the best or it has improved a lot, but that's still 1 major complaint.
Often hear there's some truth in it. But they broke new ground like 12 years ago. Twisted has, has, invented 12 years ago. Now the languages are all moving in or most are moving in that direction, like c sharp and Java with completable futures and Java 8 and stuff like this, but Twist it broke new ground. And if you break new ground, you, you probably don't get the most polished solution. And this is the point where, where Python, I think now has Python 3.5. I'm, that's really cool. I think this is a major breakthrough in, in, in the language because it brings all that async goodies into the, into a nice, syntax, nice approachable syntax. So I, I would recommend personally to write application components or apps in, in, if you, if you're using Python, think about making it Python 3 0.5 plus only. And if you can do that, then you can use all the goodies and that's, that's great.
So maybe 3.6 because they have, that's the last gap closed in 3.6 is, is async iterators into context managers. So that was the last missing part. And, Yeah, that that's cool. So I think for app developers, they really should consider cutting off old ropes. Just make it 3.5 plus or 3.6 plus and use use the language, use the new syntax. It's just great. It's compared to, if you, if you compare it to what C plus plus has or what also was it, what Java has it's it's much, much better. So the closest is probably c sharp, but even that it's, of course, Python is better than c sharp, but, they are pretty close.
[00:31:32] Unknown:
And then going back to what you're saying about having some sort of authorization rules in the router itself. I'm wondering if the crossbar router encapsulates any other business logic or if most of that is pushed to the edges of the system and managed at the client level. It's definitely pushed to the we, that's a paradigm. So we say no business logic in the router.
[00:31:54] Unknown:
So even if you have, like, a custom authorization need, then you can plug in your custom authorizer into crossbar, but that's still not code in crossbar. It's we should that should be separated and we have that ability. You can you can have a custom authorizer for your app and still have that separate from from the networking or cross path from the routing code. So, no, we you shouldn't have any business logic in in in the router. That was also a lesson from the first version of WAMP. It doesn't doesn't scale. It's you get into trouble. It's all into Weave then, and no. That's
[00:32:34] Unknown:
And what are some of the typical kinds of applications or development paradigms that cross bar is designed for and enables?
[00:32:42] Unknown:
I think that's that's probably, 1 of the most important questions. I would say there are 2, words describing or 2 approaches describing our best support paradigm. That's it's microservices and the probably a little bit older components in interfaces. So if you have the idea that you want to structure your application from components that implement defined interfaces and have those components distributed potentially over multiple systems. That's the ideal situation for, for Womp or Womp is designed for that situation. And let, if you call it that microservices, I don't care. So call it microservices, call it components or interface. That's for me in a way interchangeable.
And if you want to build your app like this, then I would say 1 piece build for that cross border build for that. We should deliver. If we don't, that's the bugger north side.
[00:33:42] Unknown:
Yeah. It seems like at least in your documentation and marketing material is also a fairly strong focus on Internet of Things devices, which again, good plays into the idea of microservices and componentized logic, but also branches out a bit as well into the physical
[00:33:57] Unknown:
realm. Yeah. It's kind of funny. That's it's that's a strategy or, more like a company focused strategy. It's like with cross where we have users from like, real time trading platforms in the web for cryptocurrencies. So this is, this is cool and exciting stuff, but this is this, and cross play is, is great for that kind of thing because you have that real time capabilities right into the web browser. But it's not the focus for us as a company because for us as a company, the it's the focus is on that industrial applications, which doesn't mean crossbar doesn't work for other things. That's that's just our focus, because we think that will be really, really big wave. If you if you look into computing history, it's what was always like the bigger, catheckery of machines being re not replaced, but the the devices where the biggest count or the biggest, market is is, they are usually in the technical leader, like mainframes, and then there came mini computers and then PCs and mobiles. And now that we have the Internet of things coming, so that will be a really big wave. And, and the other thing is that it's inherently, most of the time in another things, applications are inherently distributed. So if you have, like a vacation planning app in a company, it's cool to have a build like microservice based. It's probably better in any way, but you can still build it as a monolith running on 1 server and that's it, but you can usually build. Yeah. So you can choose, but with internal of things apps, you can choose that you have 3 plant locations.
That's just a given fact. You can't discuss it away and come up with the monolith, that only works on 1 location. So the need to have a seamless connectivity is much bigger inherently distributed applications than in app apps where you still can choose. So this is 1 aspect in there.
[00:36:06] Unknown:
And what are some common application development paradigms that you see that you think would be better suited for crossbar and the WAMP protocols, even things like web development or backend systems?
[00:36:18] Unknown:
Well, backend systems is is definitely 1 if you if you have, like like that mentioned distributed need. Another area probably if we had users that are using crossbar strictly for inter process communication on a, on a single device. So they were like, oh, we have debuzz. That's cool. But we debuzz isn't that maybe integrated with the browser side of things. And so they were looking for better IPC, basically. And they were and they came up or using are using crossbar for this. So it works even probably that's another area. And, well, the real time web concept have that need. Basically coming from the real time side of things. What else? Well, we had, you know, from another users, that is deploying crossbar in a data warehouse to hook up the different parts there.
So, but it's, it's probably fair to say it's more like a backend thing of, so of course you're all we know it's not correct. It's we were looking at like frontend components also like at, like at any other component, like take a browser front end from our point of view, a browser front end is just another component. It's just another microservice. The fact that it's, it's a interface to users, a detail. So well, for, for all, for us, the distinction front and back end is kind of dissolving in a way. And that was also the the original vision. So don't think about client and server front and back end. Try to think more like interacting services.
So that's at least our our perspective.
[00:38:00] Unknown:
What are some of the most interesting or surprising uses of crossbar and the WAP protocol that you've seen or implemented? Then I hand it over to to Alex. Maybe I can come back then in the end and then yeah. Okay. Yeah. I tried to tried to getting into the call on another machine. It also didn't work. So don't know gremlins today, apparently. Yeah. I saw that. So I was just asking to be asked, what are some of the most interesting or surprising uses of cross that you've seen or created?
[00:38:27] Unknown:
Well, I think the most interesting use that I've seen is a company from, I think, Slovenia that was using Womp and Crossbar to control old analog synthesizers. They had like a refit kit for those that you replaced the knobs with motorized ones, and then that was controlled by a Vomp. That's pretty cool. Yeah. Tobias also, mentioned the Bitcoin trading. And then also surprising, I'm not sure if it's a good thing or something, but somebody once posted something that he actually does the entire delivery of, his websites via Womp now. It's basically he loads an initial small JavaScript piece and the rest all goes via Womp which kind of raises the question why not using the established mechanisms, including caching and stuff on the way. But it's certainly interesting that. Yeah. I guess that's probably why. It's just for novelty factor because sometimes just because you can is a good enough reason. I had that impression. It was a yeah. I think he just went down a rabbit hole there, but maybe it actually brings some advantages in his particular situation. It's difficult to tell without knowing all the facts. And we briefly touched on earlier the aspect of the Internet of things for that crossbar enables. What are some of the most interesting sort of distributed physical device applications that you've seen or heard about? Well, there's in Finland, there's some building management where they're collecting that.
And apart from that, it's, there's also our showcase project or not really showcase but a friend who's selling, like has a drinks delivery business wants to move things into the 21st century to get an advantage compared to the bigger players in the market and we upgraded a small fridge dispenser unit to be remote controlled and to have remote access to the current fill level. And we are about to deploy that in the coming weeks to a co working space around the corner to test that out under real life conditions, which is our first actual, not non demo application.
[00:40:25] Unknown:
Yeah. I saw the, blog post about that on your site. It definitely looked pretty, interesting and a somewhat amusing application for technology is being able to say, okay. You know, how full is my, vending machine so that I I mean, that's 1 of the promises of technology is that it increases efficiencies so that you don't just have somebody blindly going around to various locations to
[00:40:51] Unknown:
Yeah. Yeah. I I think it's also a promise that will come true anyway. I mean, with any new technology or any new wave, it's always the question where are the applications that we can't yet foresee and there will be those. But the big thing about the Internet of Things will just be that those everyday things get improved. In a lot of cases probably in very subtle ways but I mean if you run a business like that, a 3% increase in efficiency is already, I mean, can make the difference between you staying in business and closing up shop. Absolutely. And what do you guys have planned for the future of Crossbar? Were there some things that people can look forward to coming down the road? Well, we are working on the commercial offerings now. So the development of Crossbar, the open source project, is going to continue, obviously, and there will be improvements and there will be feature additions. But the main focus at the moment is on the commercial add on service. And the features there are basically to get out of the limitation of crossbar, the 2 main limitations of crossbar, the open source thing at the moment for business deployments, which is that it's a static configuration file, read at startup once and then you need to restart to reconfigure things. And the second thing is that it's a single instance, a single node that it doesn't really communicate with other nodes. And so the management service, we have an alpha test running that gives access to the configuration API so that you can reconfigure at run time remotely via our service. And then there's also work on what Tobias was already talking about, the clustering and federation of routers. I mean those are the 2 main areas and then there's a long list of things to have monitoring with predefined and then key metrics and metrics that you can define. We're also looking into tracing of interactions.
So because with these distributed applications, the main thing is to then spin up 4 or 5 components and something happens and you're looking at 5 different log outputs in a couple in a browser, then in the shell and you need to coordinate all of these to get an actual picture of what's happening. And we want to make that easier and that you can say Okay, here are the sessions. Give me everything that's happening between these and possibly in a nice graphic representation. Or what's happening on this? Or like who is calling this and when? And then trace that back to make that easier.
[00:43:14] Unknown:
Yeah. I, work in operations. And 1 1 of the perennial pain points is trying to debug applications via log out, particularly when you don't necessarily control the contents of those logs as to, what kind of information gets dumped out. So that that that can definitely be difficult, particularly when you're talking about distributed systems and multiple different pieces that are interacting to complete a single request. Yeah. And then the log output might I mean, the components will typically run on different machines, less in development, but definitely in deployment. Then how do you even gain access to the logs there? So that's going to be a centralized thing that collects everything, the logs plus additional
[00:43:51] Unknown:
functionality on top to ease the coordination of things and dig deeper.
[00:43:55] Unknown:
So are there any other topics or questions that you think we should talk about that we didn't cover yet?
[00:44:01] Unknown:
No, not really. I think I mean, Tobias covered the ecosystem aspect, which is really important to us.
[00:44:08] Unknown:
And no. I I don't know. Did you have a look at the areas of confusion yet? Did that come up? We did not yet. Yeah. Let's go into that a bit. So what are some of the areas that people get confused about when they first start trying to implement things using crossbar in the WAMP protocol?
[00:44:23] Unknown:
Well, the thing is it's it's also a system with several parts. You have the router. You have the client libraries. You have the protocol. The relationship between these 3 aspects is kind of difficult to grasp initially quite often. And, I mean, our documentation has improved a lot. There is now documentation for pretty much anything that Crossbird does, but finding things is still problematic. I mean, there could be more cross linking, there could be more introductory things. And so people will ask things and then you the answer is just a link to a particular documentation page. And and also the that you need a certain bit of background, I think, to really be able to get into using Crossbar or a software of this kind. And then you often have people who get into this. Okay. There is a need for doing this and don't have the basic concepts and there's a lot of confusion there. I mean, you get questions and then you need to ask them, okay, let's take a few steps back because the concepts in that question don't make sense. So it's less a specific thing than this general thing. It's a specialized to comprehend and understand,
[00:45:28] Unknown:
you know, what are the use cases for? Yeah. Distributed systems can definitely be very difficult to comprehend and understand, you know, what are the use cases for it? What are the breaking points? How do you deal with network failures? Because particularly if you're coming from a, you know, somewhat more traditional development background, you're used to doing your development on your local machine, and you don't necessarily have to deal with all of those vagaries of, you know, what happens if this system isn't available? How do you manage graceful degradation of services? Things like that. Yeah. And even even before that, I mean, 1 of the initial questions often
[00:46:00] Unknown:
is, why do I need a router? I just want to connect these 2 things. Why doesn't that work just peer to peer, basically? And then Mhmm. You go back to yeah. It's with the decoupling and everything that again, basic principles and basic yeah. Basic principles for this area. So is there anything else that you think we should cover while we're on the call? I don't know. From my side, not really. I'll ask Tobias anything. Anything else you want?
[00:46:24] Unknown:
Not. Tobias also. I think the questions pretty much covered a wide area of what we're doing. So Great. Okay. So I'll ask you guys to send me your preferred contact information. I'll include that in the show notes for anybody who wants to keep up to date with what you guys are doing both personally and at Crossbar. Mhmm. And with that, I'll take us to the picks. So for my pick today, I'm going to choose, Logan, the latest movie that came out in the X Men franchise. It's a really well done movie. It's a pretty significant break from a lot of the sort of stylistic aspects that you're used to from some of the other X Men movies, and it tells a much more, sort of personal and realistic and nuanced story than what we're used to with a lot of the superhero films. And there's some pretty phenomenal acting, a lot of really great subtext in the movie, so I highly recommend that for anybody who's even remotely considering it. I thoroughly enjoyed watching it and would likely watch it again.
[00:47:19] Unknown:
Awesome. I was wondering about pics whether that's something Python related where I, as a non Pythonist myself, I wouldn't really have anything to contribute. Ah, so media, the current
[00:47:30] Unknown:
It can be absolutely anything. It can be Python, it can be movies, be a book you read. It could be what you had for dinner last night. It's literally
[00:47:37] Unknown:
anything that strikes your fancy. So, yeah, with that, I'll pass it to you, Alex. Do you what what are your picks for us? Pivotal Tracker. I'm just Okay. Starting to really, really use that again. It just makes it's an initial effort that you put all your tasks in 1 place and it's that you don't just keep additional notes around. But it just makes life so much easier to know that there's 1 place where everything that you need to do is stored and you have it easily accessible, easily to easy to reorder. And it's a very nice implementation. I mean very low overhead compared to other things. We tried where it's 15 clicks to start a new task and you just want to constantly,
[00:48:14] Unknown:
I don't know, throw things at the screen because of the interface speed. And this works in the browser, works nice, works quick. Yeah. Just about the right amount of features I I find. So Yeah. Yeah. Project management tools are difficult to find the right fit for it. Where I am right now, we do most of our work open source, and so we use GitHub issues for a majority of what we manage for projects. And a lot of times, it's just not the right fit. So it's a constant conversation as to what is the best tool that we should be using for this. Yeah. And, I mean, GitHub issues. Yeah. For software issues, okay. But if you have other stuff, no, then it doesn't have enough features. Not really. Right. Yep. All right. Okay. I'll hand on to Tobias
[00:48:55] Unknown:
for his
[00:48:56] Unknown:
okay. My pigs. I I was thinking that would be, my Python pigs, and therefore, I have a small collection of Python pigs. But sure. It can be absolutely anything. Python is definitely a valid choice. So. Okay, great. I'm lucky with that. Yeah. Well, first choice is of course Pypie. So if if anyone out out there, hasn't heard about that or hasn't used it or hasn't, give it a try, do so. That's it's really, it's an awesome project. The those guys are really incredible smart, and it's it's astonishing what they did and what they accomplished. And it's it Piper really deserves to get into the mainstream and to get into production more often. So that's my first pick. My second pick is t txtorcon.
I'm not sure I can post those links in the end probably or send it to you via email. Txtorcon is an, a Tor library for Python, which is which works on twisted. And it's it's, written by Mike Warren, which is a who's a developer with us, a core developer with crossbar. And, I think that's a cool library. It's if you any if you do want to do anything with Tor in the Python world in async land, check out Tx Torcon. And, well, my other picks. Anything from Brian Warner. So if you're into crypto stuff and Python, would highly recommend Google up Brian Warner and GitHub. You will find his page. I will send the link to. He's doing so many great things. Well, my last 2 picks are just small libraries, but very usable if you're doing I've I've lately, started to implement a command shell for crossbarfabric, which is our upcoming, cluster manager. And I had a need for a great command line parsing and, a great, REPL loop, basically. And there are 2 great libraries, Qlik by Armin Roanacher, which is, known probably most for Flask. Its hook line is composable command line interfaces, and that's exactly what it does. It's you can build complex command line interface, like think of git easily with Qlik, which is great. You can build composable command lines. The other toolkit I want to mention is Python prompt toolkit, which is also really, really cool. It's for REPL loops. You can implement custom shells in it and, combining both is a bit tricky, but it does work also. So you can combine click and and the prompt toolkit by I will send the link. It's also on GitHub. So those are my my picks. So it's more the last ones are more practical side. I want to do command lines. Check that out And, well but the overall thing, PyPI, check it out. It's cool.
Anyway, so
[00:51:56] Unknown:
thank you so much for having us. Yeah. Thank you both for taking the time out of your day to join me and tell me more about crossbar and your work with the WAMP protocol. It's definitely interesting and something that I'm going to have to keep a
[00:52:09] Unknown:
closer eye on going forward. Yeah. Cool. So thanks again. And, yeah. Well, looking forward, for more podcasts. And and, if you have Brian, that's cool.
Hello, and welcome to podcast.init, the podcast about Python and the people who make it great. I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. When you're ready to launch your next project, you'll need somewhere to deploy, so you should check out linode@linode.com/podcastinnit and get a $20 credit to try out their fast and reliable Linux virtual servers for running your app or trying out something that you hear about on the show. You can visit the site at www.podcasttinnit.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. And to help other people find the show, you can leave a review on iTunes or Google Play Music, and tell your friends and coworkers and share it on social media. The host, as usual, is Tobias Macy. And today, I'm interviewing Tobias Oberstein and Alexander Gode about crossbar dot io, a high throughput asynchronous router for the WAMP protocol. So, Tobias, could you please introduce your I guess I should probably redo that and pronounce your name properly. So used to people anglicizing my name. So
[00:01:04] Unknown:
That that's awesome. I'm that's the first time I've been interviewed by by another person who's, with the same 4 names. So my my name is Tobias, and, I'm cofounder of Crossbar. Io, of both of the project. And, recently end of year last year, we founded a company focused on that product or the project. And the co founder is Alexander Goede as, as he mentioned it, hopefully he will be able to join in. And, Yeah. Well, what should I say? I'm into computers since a long time. So my, my originally introduced, was, by my stepfather who brought, some single board computer to our home. That was, I was like 12 years and then got into hacking.
And, well, then I've professionally, I've been doing a lot of consulting work for big companies in in data warehousing and other stuff. But in the, in the last couple of years, I've been much into Python. And then I've, we've started, well, probably another project besides crossbar, which is known to, to some guys probably that, which is Autobahn, which is, well, it's, it's a, it's a collection of web socket libraries and, Womp libraries. And yeah. Well, that's probably enough to for for intro intro to my person. So And how did you first get introduced to Python? I was hoping you wouldn't ask me because honestly, I forgot. I was trying to, to, reconstruct when the year when I've started to use Python, something thing around 2, 002 because at that point I was still in a master theaters and, that was mainly C plus plus but I, I wanted to wrap that making it, easier for, for other researchers to approach. And that was the point.
I definitely know I was using Python at that point. And then I was using, other languages over the years. So I left Python for a couple of years and then came back. So now I'm pretty much using it every day and for the last couple of years. Well, I I'm, I not, I not left C plus plus completely, but, every time I'm coming back, it's like, you know, you get used to the simp, simplicity of Python and the it just works. So it's doesn't stand in the way. So I'm a I'm a heavy Python user
[00:03:35] Unknown:
since a couple of years now. So, yeah, if you could just give us a brief introduction to what is Crossbar and what was the problem that you're trying to solve when you first created it. Crossbar nowadays is positioned as a multi protocol application
[00:03:49] Unknown:
router. So what's that? So the idea is, you have a distributed application consisting of components, or you have, application built around the microservice microservices approach. Approach, then then you have a need for how do those components services communicate. And, crossbar provides, infrastructure for the communication between application components and, and microservices. So, that's the application router part and the multi protocol part is, as, as you mentioned, it's, it's based originally based around a WAMP, which is based internal web socket.
And that protocol was also started by us. It provides 2 application messaging patterns. So remote procedure calls and publish subscribe and, Yeah, well, so crossbar aims to provide a high level, communication infrastructure for apps and has the aim for seamless connectivity. So that's, that's our goal. Seamless connectivity in our view means crossing languages. So it's not Python only and crossing network and system boundaries. So that's, that's the, the scope and the original problem we had was we, we wanted it that's years ago, some years ago we wanted to build a mobile app and, and the app should talk, should have talked to your back end end and at the same time have a browser front end talk to that same back end. So then you have a multi language scenario or we had that multi language scenario and we were, we've been looking for a protocol to make that happen, to connect those front ends to the backend. And not only do that, but have a real time, ability for the user interface. So being able to update the user interfaces in real time from the backend. And at that point, that was like 2011 or 12.
We've been looking around for suitable protocol and didn't found anything or basically was all hacks. So the, the, the web part or a browser front ends, there was combat and other hacks to get real time capabilities. But at that point, WebSocket started or the, the ITF started a WebSocket for a working group to, to come up with a new protocol for the web, which is bidirectional real time. And that is WebSocket. That was the point where we started to where, well, crossbar started to develop. So, and nowadays we that old project, that old app it's long quit. So we're not we're not, developing that anymore. But now the focus is just on that problem, the communication problem.
[00:06:36] Unknown:
And I noticed when I was going through the documentation to research the show, I noticed that you had started an actual IETF draft protocol proposal for the WAMP protocol. So I'm wondering what the status of that proposal is and if you ever made it over through to having be an RFC. Yeah. That's, it's a very good question
[00:06:55] Unknown:
because well, the status is like this. The working group for WebSocket, has fulfilled the original goal, the original chapter. I think they call it chapter or something. And, so the chair of the working group, not close to working group because it reached that goal. And so we are looking for new well, to develop an RFC in the I t f you need a working group because you need exchange between different implementers. And, we've lost, that is kind of not stalled, but in the, in the waiting queue for us, it's we have so much to do. So we we want to to, to take up the work again in within the ITF, in that working group to make it into a proper RC, but that's still a lot of work.
So currently we, we just don't have the time to, to push that forward, but it's on the we definitely want to because, I I think it would we we already have a lot of, alternative implementations, which is you normally the the the biggest hurdle to get into to get an RSC because the ITF has that policy of you need 3 independent implementations. And so that's normally the biggest hurdle or barrier, but we already have that. So it's, it's a lot of editing and publishing work. I've been in the work in the ITF working group for web socket was like 1 and a half years intensive discussions. And it's, it's really a lot of work. You can, can't overestimate that if you, because you get into exchange with very, very smart people.
And obviously they have, hard questions and hard critics and going through this whole process is a lot of work. So we want to do it, but we don't have the time right now. So if there's anyone who's, saying, oh, I'm into network protocols, probably you want to have, have a chance to be, like coauthor or co editor of such an IRC. That will be awesome. So if there's anybody else out there, just get in touch with us. So
[00:09:12] Unknown:
What prompted you when you were first drafting
[00:09:23] Unknown:
Oh, that Oh, that that's a very good question. I I I think it's, me, I was like me personally, I wouldn't use anything which isn't an open standard or which tries to lock me into something. As a as a developer, I was like, let's we there isn't any, any point in making it closed or developing a closed protocol, I think nowadays. So it, it needs to have, alternative implementations. And to, to do that, you have to have an open protocol. So we we said in the beginning, and we also said, but, we don't want to lock developers into our stuff. So there should be different alternative implementations for Womp. Not only so if you say I want, I like WAMP, but I don't like the crossbar or Autobahn stuff. That's cool. You can you can use a non cross bio WAMP router and you can use non auto bond client libraries. And, we totally support that.
Of course, we say our implementations are great and and and we're good, but, you you shouldn't be locked into. So, I'm pretty much convinced, even if I was talking, like some guy once asked me, why are you like like marketing, alternative implementations? Doesn't that hurt you, sir, your own company? And, I'm pretty much convinced. No, it doesn't. It it's on the law in the long run, it's necessary. So you should be able to have, not being locked in. So that was the idea from from the very beginning. And then it's a thriving ecosystem. So there are new client libraries and and also router, implementations coming out, all the time. So just in the last days, it was, another C plus plus client implementation announced. So yeah. Alexis.
So, people all like the the the, days before or weeks before was like Pearl was another we now have a Pearl client library implementation for WAMP, which is awesome. Awesome. I'm I'm not using WAMP, or Pearl obviously, but, others are doing and then, and those are, so the gaps are closing. I I think it's a healthier ecosystem. It's, it's growing. And, well, may maybe that's also an important point. We have a dedicated webpage where we, where we really, well show have, have a list of implementations and we don't have a policy whatsoever. If you say you, you, you have a WAMP implementation regardless client or router, and you're aiming for being compatible, then we will put your stuff, link your stuff on the WAMP website.
So there's no discrimination in any way. The only thing we say, if, if you're trying to modify the protocol, that's also cool, but then don't call it 1 because that would confuse probably confuse users then. So we we reserve, we still reserve a last word in, like like, you using the womp trademark or that word, just to avoid confusing users. But you could even frog the protocol if you wanted to, which didn't happen today or out of up to up today, but, you could. And yeah, well, so I, I think, and there's also mailing list, which is, has quite some healthy activities. So yeah, I would say language wise gaps are now pretty much closed. Of course they're, like esoteric or I think we even have a Haskell implementation or something.
So there's still some gaps left, but they are closing. So
[00:13:04] Unknown:
That's That's how you know you've made is when somebody takes the time to actually implement your protocol on Haskell.
[00:13:10] Unknown:
Yes. We have have it. I I see it has to go, so it's done already. So, but, Yeah. And and to your point earlier as well about
[00:13:21] Unknown:
having the multiple different implementations and competing implementations as well, the rising tide lifts all boats, and it's a similar situation to the conversation that I had with the folks at Sentry where they have a hosted offering, but they also offer the exact same product as an open source self hosted site. And because of that, I think is what leads to a lot of their actual usage Because as you said, there's no lock in. People can start using 1 implementation and then easily shift to another 1 without having to completely rearchitect their entire application. Definitely. Definitely. So for example, with the Audemann libraries, we try to bring over the developer
[00:14:01] Unknown:
experience over different languages or as close together as possible. So that bit, but that's only 1 like which leads to certain style in, in the Autobahn C plus plus for example. And, but there definitely is room for different approaches. So we they're competing c plus plus implementations, which are optimized in a different way. For example, qt or not using boost. So the this last library in c plus plus they had the issue like, c plus plus boost is a, is a monster. So you don't want to, it's cool, but you don't want to have it always. So there's room for, for different, even in the same language. So there, even if, if you take just C plus plus or JavaScript, there's also alternative JavaScript, 1 implementation.
So I'm, I'm a totally total fan of that because, well, you have choice. Developers should have choice. And, if, if there's another 1, the other guy coming around with a better JavaScript experience than Autobahn. Js. So that's cool. Why not? So we open to that. That's that's I think that's cool.
[00:15:10] Unknown:
So the reference implementation and crossbar was written in Python. So I'm curious that given the goals of the WAMP protocol is to be fairly high throughput, I'm wondering why you ended up choosing Python as the,
[00:15:23] Unknown:
language of choice for the reference implementation for something that is intended to be high throughput and low latency, and also how you manage to achieve those goals while using Python? Yeah. That's a very nice question because I love it because I love like talking to the Java guys, Java people. I'm still using Java, but they're always like, oh, we have that cool JVM hotspot. Nothing is like that. And stuff like this is all, this is old. This is like the world is it's not the same, like a a couple of years ago. And people probably haven't, or not all developers have like acknowledged that tiny scripting languages like Java script or Python have come a long way also performance wise. So I'm, I'm coming from the C plus plus side of things. And so I I'd I'd say, okay, I'm I I know how to do bare metal stuff, but, Python offers just such a a bigger or better developer productivity compared to like in particular C plus plus so that's a main point for us productivity.
For example, crossbar, if you take everything, Autobahn, it's like 50 ks lines of code And it's for that. It's doing a lot. And if we had it implemented in C plus plus it would be multiple of that. And we we couldn't have pulled that off as a small company or with a handful of of developers. So the productivity is, is much better. Like performance wise, if you, you can, like in, in, in Python, you have different implementations. So there's C Python, which is the classic interpreter. But nowadays we have PyPy, which is a tracing chip compiler. And this is what we use in production.
So it's the crossbar and Autobahn in production. We usually run it on PyPI. So it's much faster than C Python because it's, it's, compiling to native code in like Java, the JavaScript engines and the browsers. And, that's a major point performance wise. And then we have, like native tiny bits of native code interface. Like for example, in the, in the web socket implementation, you, you have to have a masking and that's, you can accelerate that still accelerate over getting it faster than the JIT can do, but that requires like vectorized code and stuff like this. So it's, we don't have, I'm pretty sure I would take bets if, if someone is, is like, wants to beat crossbar its own game. I'm pretty sure it will be competitive. So yeah. Well, what in that area, what else could I say? The the commercial maybe that's interesting. The commercial offering of crossbar has, has native code acceleration, which which is, yeah. Well, adding, another level of performance. But if you if you'd ask me if, now in in looking back, would we rewrite crossbar in like another language?
And, I was asked at, at 1 point by some guy and the answer is I would only consider 1 other language and that's rust. So, but that's not on the on our road map. So and we don't have a need right now. So it's it's fast. What we want to what we need to add is multi core, multi node support, but that's not a question of implementation language. It's more a question of getting all the bits talking together. And so no. We don't have a problem and I would take any challenge in beating cross by its own game, like from the Java guys.
[00:19:10] Unknown:
So could you share a bit about how Crossfire is architected and some of the underlying technologies that you're using for allowing it to manage that high throughput and also the, sort of highly distributed nature of the client traffic as well?
[00:19:24] Unknown:
So that's actually 2 questions. So internally, Crossbar has a multi process architecture. So that's, a crossbar node always starts with a controller process and then forks off worker processes. So we have the ability to scale on multi core by using multiple processes. So that's, that's a major point. And an important aspect in this is that we also using Womp internally. So the, the different worker processes in a crossbar node, they are talking to the node controller process also using WAMP. That's probably that recursive nature. Alex has recently told me that it's took him quite some, some time to get behind that story, but it's that's we using wamp internally to in the process communication for the in the process communication. We're also using Womp for the upcoming, management, cluster manager. And, that's also important part maybe that, or that touches that second part of the of your question, highly distributed sets of clients. How we how do we treat that? That's really our vision. We we think it'll, applications in that area of the Internet of Things or, industry 4.0, it's called here in Germany, is you have inherently distributed application with you have different locations, factory floors, you have a cloud back end ERP systems, you have mobile devices and all that. And our reason is not to connect all those end points to 1 crossbar node running in the cloud, but to have also a network of crossbar nodes also distributed.
And that answers or that touches that low latency questions. Because for example, imagine you have a machine on a, sir, on a certain factory floor in the location a, probably that machine is wrapped as a warm component and should be able to talk to a backend on the local location on that plant. But it should also be able to talk to a cloud back end. So you have a low latency for the local communication to the local backend running on that location a. And you have a necessarily bigger latency if that wants to communicate. The shop flow wants to communicate with the cloud backend in, in, in the data center. So it's not a question And you want to have both, you want to have the ability to, to let it talk to your cloud, but also have low latency, high throughput locally. So you want to have a it's it's called like in in there's a term like edge cloud, FOC computing.
So the idea is to extend the cloud right into edge locations. I have to say we're we're still not there. So we that's, I'm I'm talking now about the vision where we want to go with crossbar. It's not ready today. What we are doing today for in the latency area is, like, for example, for our own demo instances, we're doing latency based DNS. So if a client comes in and wants to resolve the host name for crossbar, it's directed. The DNS is resolved based on the latency to the crossbar node. So if you're in the US and the opening or demo page, you should be directed to an US instance where you get lower latencies, compared to as if you were connected to the European instance. So we have multiple instances around, the globe and you're directed to the at, already at the DNS level to the nearest instance.
So that reduces latency because like going from here, Germany to west coast us is like 250 milliseconds, round trip, or 200 to 215. Possibly that's just physics. It takes it's like a couple of 1, 000 kilometers. So you can't, trick, speed of light. So that's basically, and that works today. So you can, we using route 53 AWS and it has a cool feature of latency based DNS. You can do that today, but, the other story is still the bigger 1 that distributed network of routers, and we are not there yet. That's a vision, and we're working on that.
[00:23:49] Unknown:
Yeah. There are a lot of complicating factors for that, not least of which is service discovery for being able to figure out where the destination point is for the remote procedure call, for instance, and figuring out what network of routers you need to transmit the message through to be able to reach that destination point. So
[00:24:07] Unknown:
But but that's that's just an very interesting aspect you're touching on the, on here because okay. Yeah. That's absolutely right. We, we have to crack that nut and that's part of the problem, But the whole issue of service discovery, like, if you're in the in the Http rest world, if you're doing microservices using the rest patterns, then all your microservices are servers, our web servers. And then you have the problem. If you want to call that, how do you discover it? And the whole, category of, of, of apps is, or is, is targeting that problem. It's called API gateways. Like, NGINX has, has some product in, in there, but, other other companies also like works like this, the, the service, the HTTP rest server registers itself at that gateway API gateway, and the gateway remembers where's the implementer of, of that servers. And then if your HTTP rest request comes in, that is, rooted to using that information to the right web server endpoint. And we don't have that whole issue in crossbar because WAMP has not only a RPC, but what we call rooted RPC.
So the the colleague side already tells the 1 prooter or the 1 prooter already knows where the, the colleague resides at the point of problem disappears with Womp. You don't have the service discovery problem. You the caller just calls, 1p u, URI or the procedure is identified by, by 1p URI and the 1p router already knows where how to forward the call. So this is and it's also the biggest difference with we we had WAMP has has had in history 1 major upgrade from 1 version 1 to 2 and okay. We don't expect the version 3, so that's, but it was a big jump. And the, and the main point in there was we've moved from RPC to rooted RPC.
So originally 1 1 was like, you had to implement your procedures as part of the router. It was, was like very inflexible and that was the biggest change. We split that out. And the nice side effect is we don't have the whole issue that you have in HTTP rest with service discovery. That's a nonissue
[00:26:34] Unknown:
for us. Yeah. And it also seems that that would help prevent the issue of the right message ending up at the wrong destination so you don't have to worry about potentially leaking sensitive contents of a message because you know exactly where the message is going to end up at the destination point without having to worry about it being broadcast and then Yeah. Relying on the endpoints to filter earlier. Yeah. Yeah. Yeah. No, that's exactly the case. So, from the router point of view,
[00:27:02] Unknown:
client components are untrusted in the sense they have to authenticate. And then, the router knows, has assigned an a role. So we have role based authorization in crossbar and then you have the ability to to configure the permissions in the router and the router enforces those permissions. So you can say on the 1 URI, the 1 role is allowed to subscribe, but not to publish. And then he even can have a publisher only allowed to be able to publish, but not to subscribe to its own events. So and the subscriber side, you can you can make it only a subscriber and disallow any publication. So that allows a quite fine grained, control over the internal also the internals of the application. So you can, like, like, enforce rules internally, not only to the outside world. So
[00:27:56] Unknown:
My understanding is that at least in the initial implementation, crossbar was based somewhat heavily on Twisted for handling asynchronous requests. And I'm wondering, 1, how you're leveraging Twisted, and then, 2, what your view of the asynchronous landscape in Python great question. It's okay. They have multiple aspects. Okay.
[00:28:20] Unknown:
To a cross bar itself is, is using twisted and it's using, in particular, it's using Autobahn in, in its twisted flavor. So Autobahn Autobahn Python supports both twisted and asyncio as the underlying network framework. So you can you can choose. But in crossbar, we've chosen twisted and, and that's, there are multiple reasons. 1 reason is we we wanted and still want to support run, running crossbond on Python 2.7. And then the more important point, Twisted is much broader in scope than asyncio also. It, it brings a lot of protocols with it. It brings a web framework with it. And, so we using that. And for example, it has first class process support, which is trick kind of tricky with if you take in signals and, and stuff like this, getting that right, isn't easy. And it's it's, they've done so much work over so many years, the twisted guys. It's it's great. It's totally underrated. It's probably, the documentation isn't the best or it has improved a lot, but that's still 1 major complaint.
Often hear there's some truth in it. But they broke new ground like 12 years ago. Twisted has, has, invented 12 years ago. Now the languages are all moving in or most are moving in that direction, like c sharp and Java with completable futures and Java 8 and stuff like this, but Twist it broke new ground. And if you break new ground, you, you probably don't get the most polished solution. And this is the point where, where Python, I think now has Python 3.5. I'm, that's really cool. I think this is a major breakthrough in, in, in the language because it brings all that async goodies into the, into a nice, syntax, nice approachable syntax. So I, I would recommend personally to write application components or apps in, in, if you, if you're using Python, think about making it Python 3 0.5 plus only. And if you can do that, then you can use all the goodies and that's, that's great.
So maybe 3.6 because they have, that's the last gap closed in 3.6 is, is async iterators into context managers. So that was the last missing part. And, Yeah, that that's cool. So I think for app developers, they really should consider cutting off old ropes. Just make it 3.5 plus or 3.6 plus and use use the language, use the new syntax. It's just great. It's compared to, if you, if you compare it to what C plus plus has or what also was it, what Java has it's it's much, much better. So the closest is probably c sharp, but even that it's, of course, Python is better than c sharp, but, they are pretty close.
[00:31:32] Unknown:
And then going back to what you're saying about having some sort of authorization rules in the router itself. I'm wondering if the crossbar router encapsulates any other business logic or if most of that is pushed to the edges of the system and managed at the client level. It's definitely pushed to the we, that's a paradigm. So we say no business logic in the router.
[00:31:54] Unknown:
So even if you have, like, a custom authorization need, then you can plug in your custom authorizer into crossbar, but that's still not code in crossbar. It's we should that should be separated and we have that ability. You can you can have a custom authorizer for your app and still have that separate from from the networking or cross path from the routing code. So, no, we you shouldn't have any business logic in in in the router. That was also a lesson from the first version of WAMP. It doesn't doesn't scale. It's you get into trouble. It's all into Weave then, and no. That's
[00:32:34] Unknown:
And what are some of the typical kinds of applications or development paradigms that cross bar is designed for and enables?
[00:32:42] Unknown:
I think that's that's probably, 1 of the most important questions. I would say there are 2, words describing or 2 approaches describing our best support paradigm. That's it's microservices and the probably a little bit older components in interfaces. So if you have the idea that you want to structure your application from components that implement defined interfaces and have those components distributed potentially over multiple systems. That's the ideal situation for, for Womp or Womp is designed for that situation. And let, if you call it that microservices, I don't care. So call it microservices, call it components or interface. That's for me in a way interchangeable.
And if you want to build your app like this, then I would say 1 piece build for that cross border build for that. We should deliver. If we don't, that's the bugger north side.
[00:33:42] Unknown:
Yeah. It seems like at least in your documentation and marketing material is also a fairly strong focus on Internet of Things devices, which again, good plays into the idea of microservices and componentized logic, but also branches out a bit as well into the physical
[00:33:57] Unknown:
realm. Yeah. It's kind of funny. That's it's that's a strategy or, more like a company focused strategy. It's like with cross where we have users from like, real time trading platforms in the web for cryptocurrencies. So this is, this is cool and exciting stuff, but this is this, and cross play is, is great for that kind of thing because you have that real time capabilities right into the web browser. But it's not the focus for us as a company because for us as a company, the it's the focus is on that industrial applications, which doesn't mean crossbar doesn't work for other things. That's that's just our focus, because we think that will be really, really big wave. If you if you look into computing history, it's what was always like the bigger, catheckery of machines being re not replaced, but the the devices where the biggest count or the biggest, market is is, they are usually in the technical leader, like mainframes, and then there came mini computers and then PCs and mobiles. And now that we have the Internet of things coming, so that will be a really big wave. And, and the other thing is that it's inherently, most of the time in another things, applications are inherently distributed. So if you have, like a vacation planning app in a company, it's cool to have a build like microservice based. It's probably better in any way, but you can still build it as a monolith running on 1 server and that's it, but you can usually build. Yeah. So you can choose, but with internal of things apps, you can choose that you have 3 plant locations.
That's just a given fact. You can't discuss it away and come up with the monolith, that only works on 1 location. So the need to have a seamless connectivity is much bigger inherently distributed applications than in app apps where you still can choose. So this is 1 aspect in there.
[00:36:06] Unknown:
And what are some common application development paradigms that you see that you think would be better suited for crossbar and the WAMP protocols, even things like web development or backend systems?
[00:36:18] Unknown:
Well, backend systems is is definitely 1 if you if you have, like like that mentioned distributed need. Another area probably if we had users that are using crossbar strictly for inter process communication on a, on a single device. So they were like, oh, we have debuzz. That's cool. But we debuzz isn't that maybe integrated with the browser side of things. And so they were looking for better IPC, basically. And they were and they came up or using are using crossbar for this. So it works even probably that's another area. And, well, the real time web concept have that need. Basically coming from the real time side of things. What else? Well, we had, you know, from another users, that is deploying crossbar in a data warehouse to hook up the different parts there.
So, but it's, it's probably fair to say it's more like a backend thing of, so of course you're all we know it's not correct. It's we were looking at like frontend components also like at, like at any other component, like take a browser front end from our point of view, a browser front end is just another component. It's just another microservice. The fact that it's, it's a interface to users, a detail. So well, for, for all, for us, the distinction front and back end is kind of dissolving in a way. And that was also the the original vision. So don't think about client and server front and back end. Try to think more like interacting services.
So that's at least our our perspective.
[00:38:00] Unknown:
What are some of the most interesting or surprising uses of crossbar and the WAP protocol that you've seen or implemented? Then I hand it over to to Alex. Maybe I can come back then in the end and then yeah. Okay. Yeah. I tried to tried to getting into the call on another machine. It also didn't work. So don't know gremlins today, apparently. Yeah. I saw that. So I was just asking to be asked, what are some of the most interesting or surprising uses of cross that you've seen or created?
[00:38:27] Unknown:
Well, I think the most interesting use that I've seen is a company from, I think, Slovenia that was using Womp and Crossbar to control old analog synthesizers. They had like a refit kit for those that you replaced the knobs with motorized ones, and then that was controlled by a Vomp. That's pretty cool. Yeah. Tobias also, mentioned the Bitcoin trading. And then also surprising, I'm not sure if it's a good thing or something, but somebody once posted something that he actually does the entire delivery of, his websites via Womp now. It's basically he loads an initial small JavaScript piece and the rest all goes via Womp which kind of raises the question why not using the established mechanisms, including caching and stuff on the way. But it's certainly interesting that. Yeah. I guess that's probably why. It's just for novelty factor because sometimes just because you can is a good enough reason. I had that impression. It was a yeah. I think he just went down a rabbit hole there, but maybe it actually brings some advantages in his particular situation. It's difficult to tell without knowing all the facts. And we briefly touched on earlier the aspect of the Internet of things for that crossbar enables. What are some of the most interesting sort of distributed physical device applications that you've seen or heard about? Well, there's in Finland, there's some building management where they're collecting that.
And apart from that, it's, there's also our showcase project or not really showcase but a friend who's selling, like has a drinks delivery business wants to move things into the 21st century to get an advantage compared to the bigger players in the market and we upgraded a small fridge dispenser unit to be remote controlled and to have remote access to the current fill level. And we are about to deploy that in the coming weeks to a co working space around the corner to test that out under real life conditions, which is our first actual, not non demo application.
[00:40:25] Unknown:
Yeah. I saw the, blog post about that on your site. It definitely looked pretty, interesting and a somewhat amusing application for technology is being able to say, okay. You know, how full is my, vending machine so that I I mean, that's 1 of the promises of technology is that it increases efficiencies so that you don't just have somebody blindly going around to various locations to
[00:40:51] Unknown:
Yeah. Yeah. I I think it's also a promise that will come true anyway. I mean, with any new technology or any new wave, it's always the question where are the applications that we can't yet foresee and there will be those. But the big thing about the Internet of Things will just be that those everyday things get improved. In a lot of cases probably in very subtle ways but I mean if you run a business like that, a 3% increase in efficiency is already, I mean, can make the difference between you staying in business and closing up shop. Absolutely. And what do you guys have planned for the future of Crossbar? Were there some things that people can look forward to coming down the road? Well, we are working on the commercial offerings now. So the development of Crossbar, the open source project, is going to continue, obviously, and there will be improvements and there will be feature additions. But the main focus at the moment is on the commercial add on service. And the features there are basically to get out of the limitation of crossbar, the 2 main limitations of crossbar, the open source thing at the moment for business deployments, which is that it's a static configuration file, read at startup once and then you need to restart to reconfigure things. And the second thing is that it's a single instance, a single node that it doesn't really communicate with other nodes. And so the management service, we have an alpha test running that gives access to the configuration API so that you can reconfigure at run time remotely via our service. And then there's also work on what Tobias was already talking about, the clustering and federation of routers. I mean those are the 2 main areas and then there's a long list of things to have monitoring with predefined and then key metrics and metrics that you can define. We're also looking into tracing of interactions.
So because with these distributed applications, the main thing is to then spin up 4 or 5 components and something happens and you're looking at 5 different log outputs in a couple in a browser, then in the shell and you need to coordinate all of these to get an actual picture of what's happening. And we want to make that easier and that you can say Okay, here are the sessions. Give me everything that's happening between these and possibly in a nice graphic representation. Or what's happening on this? Or like who is calling this and when? And then trace that back to make that easier.
[00:43:14] Unknown:
Yeah. I, work in operations. And 1 1 of the perennial pain points is trying to debug applications via log out, particularly when you don't necessarily control the contents of those logs as to, what kind of information gets dumped out. So that that that can definitely be difficult, particularly when you're talking about distributed systems and multiple different pieces that are interacting to complete a single request. Yeah. And then the log output might I mean, the components will typically run on different machines, less in development, but definitely in deployment. Then how do you even gain access to the logs there? So that's going to be a centralized thing that collects everything, the logs plus additional
[00:43:51] Unknown:
functionality on top to ease the coordination of things and dig deeper.
[00:43:55] Unknown:
So are there any other topics or questions that you think we should talk about that we didn't cover yet?
[00:44:01] Unknown:
No, not really. I think I mean, Tobias covered the ecosystem aspect, which is really important to us.
[00:44:08] Unknown:
And no. I I don't know. Did you have a look at the areas of confusion yet? Did that come up? We did not yet. Yeah. Let's go into that a bit. So what are some of the areas that people get confused about when they first start trying to implement things using crossbar in the WAMP protocol?
[00:44:23] Unknown:
Well, the thing is it's it's also a system with several parts. You have the router. You have the client libraries. You have the protocol. The relationship between these 3 aspects is kind of difficult to grasp initially quite often. And, I mean, our documentation has improved a lot. There is now documentation for pretty much anything that Crossbird does, but finding things is still problematic. I mean, there could be more cross linking, there could be more introductory things. And so people will ask things and then you the answer is just a link to a particular documentation page. And and also the that you need a certain bit of background, I think, to really be able to get into using Crossbar or a software of this kind. And then you often have people who get into this. Okay. There is a need for doing this and don't have the basic concepts and there's a lot of confusion there. I mean, you get questions and then you need to ask them, okay, let's take a few steps back because the concepts in that question don't make sense. So it's less a specific thing than this general thing. It's a specialized to comprehend and understand,
[00:45:28] Unknown:
you know, what are the use cases for? Yeah. Distributed systems can definitely be very difficult to comprehend and understand, you know, what are the use cases for it? What are the breaking points? How do you deal with network failures? Because particularly if you're coming from a, you know, somewhat more traditional development background, you're used to doing your development on your local machine, and you don't necessarily have to deal with all of those vagaries of, you know, what happens if this system isn't available? How do you manage graceful degradation of services? Things like that. Yeah. And even even before that, I mean, 1 of the initial questions often
[00:46:00] Unknown:
is, why do I need a router? I just want to connect these 2 things. Why doesn't that work just peer to peer, basically? And then Mhmm. You go back to yeah. It's with the decoupling and everything that again, basic principles and basic yeah. Basic principles for this area. So is there anything else that you think we should cover while we're on the call? I don't know. From my side, not really. I'll ask Tobias anything. Anything else you want?
[00:46:24] Unknown:
Not. Tobias also. I think the questions pretty much covered a wide area of what we're doing. So Great. Okay. So I'll ask you guys to send me your preferred contact information. I'll include that in the show notes for anybody who wants to keep up to date with what you guys are doing both personally and at Crossbar. Mhmm. And with that, I'll take us to the picks. So for my pick today, I'm going to choose, Logan, the latest movie that came out in the X Men franchise. It's a really well done movie. It's a pretty significant break from a lot of the sort of stylistic aspects that you're used to from some of the other X Men movies, and it tells a much more, sort of personal and realistic and nuanced story than what we're used to with a lot of the superhero films. And there's some pretty phenomenal acting, a lot of really great subtext in the movie, so I highly recommend that for anybody who's even remotely considering it. I thoroughly enjoyed watching it and would likely watch it again.
[00:47:19] Unknown:
Awesome. I was wondering about pics whether that's something Python related where I, as a non Pythonist myself, I wouldn't really have anything to contribute. Ah, so media, the current
[00:47:30] Unknown:
It can be absolutely anything. It can be Python, it can be movies, be a book you read. It could be what you had for dinner last night. It's literally
[00:47:37] Unknown:
anything that strikes your fancy. So, yeah, with that, I'll pass it to you, Alex. Do you what what are your picks for us? Pivotal Tracker. I'm just Okay. Starting to really, really use that again. It just makes it's an initial effort that you put all your tasks in 1 place and it's that you don't just keep additional notes around. But it just makes life so much easier to know that there's 1 place where everything that you need to do is stored and you have it easily accessible, easily to easy to reorder. And it's a very nice implementation. I mean very low overhead compared to other things. We tried where it's 15 clicks to start a new task and you just want to constantly,
[00:48:14] Unknown:
I don't know, throw things at the screen because of the interface speed. And this works in the browser, works nice, works quick. Yeah. Just about the right amount of features I I find. So Yeah. Yeah. Project management tools are difficult to find the right fit for it. Where I am right now, we do most of our work open source, and so we use GitHub issues for a majority of what we manage for projects. And a lot of times, it's just not the right fit. So it's a constant conversation as to what is the best tool that we should be using for this. Yeah. And, I mean, GitHub issues. Yeah. For software issues, okay. But if you have other stuff, no, then it doesn't have enough features. Not really. Right. Yep. All right. Okay. I'll hand on to Tobias
[00:48:55] Unknown:
for his
[00:48:56] Unknown:
okay. My pigs. I I was thinking that would be, my Python pigs, and therefore, I have a small collection of Python pigs. But sure. It can be absolutely anything. Python is definitely a valid choice. So. Okay, great. I'm lucky with that. Yeah. Well, first choice is of course Pypie. So if if anyone out out there, hasn't heard about that or hasn't used it or hasn't, give it a try, do so. That's it's really, it's an awesome project. The those guys are really incredible smart, and it's it's astonishing what they did and what they accomplished. And it's it Piper really deserves to get into the mainstream and to get into production more often. So that's my first pick. My second pick is t txtorcon.
I'm not sure I can post those links in the end probably or send it to you via email. Txtorcon is an, a Tor library for Python, which is which works on twisted. And it's it's, written by Mike Warren, which is a who's a developer with us, a core developer with crossbar. And, I think that's a cool library. It's if you any if you do want to do anything with Tor in the Python world in async land, check out Tx Torcon. And, well, my other picks. Anything from Brian Warner. So if you're into crypto stuff and Python, would highly recommend Google up Brian Warner and GitHub. You will find his page. I will send the link to. He's doing so many great things. Well, my last 2 picks are just small libraries, but very usable if you're doing I've I've lately, started to implement a command shell for crossbarfabric, which is our upcoming, cluster manager. And I had a need for a great command line parsing and, a great, REPL loop, basically. And there are 2 great libraries, Qlik by Armin Roanacher, which is, known probably most for Flask. Its hook line is composable command line interfaces, and that's exactly what it does. It's you can build complex command line interface, like think of git easily with Qlik, which is great. You can build composable command lines. The other toolkit I want to mention is Python prompt toolkit, which is also really, really cool. It's for REPL loops. You can implement custom shells in it and, combining both is a bit tricky, but it does work also. So you can combine click and and the prompt toolkit by I will send the link. It's also on GitHub. So those are my my picks. So it's more the last ones are more practical side. I want to do command lines. Check that out And, well but the overall thing, PyPI, check it out. It's cool.
Anyway, so
[00:51:56] Unknown:
thank you so much for having us. Yeah. Thank you both for taking the time out of your day to join me and tell me more about crossbar and your work with the WAMP protocol. It's definitely interesting and something that I'm going to have to keep a
[00:52:09] Unknown:
closer eye on going forward. Yeah. Cool. So thanks again. And, yeah. Well, looking forward, for more podcasts. And and, if you have Brian, that's cool.
Introduction and Sponsor Message
Interview with Tobias Oberstein and Alexander Gode
Tobias Oberstein's Background and Introduction
Introduction to Crossbar.io
Development of the WAMP Protocol
Open Standards and Ecosystem
Choosing Python for Crossbar.io
Architecture and High Throughput Management
Service Discovery and Distributed Systems
Role-Based Authorization and Security
Twisted and Asynchronous Programming in Python
Business Logic and Router Design
Applications and Development Paradigms
Interesting Uses of Crossbar and WAMP
Future Plans for Crossbar
Common Areas of Confusion
Picks and Recommendations
