Path To Citus Con, for developers who love Postgres

From developer to PostgreSQL specialist
===

CLAIRE: This is your official welcome to Path to Citus Con, the podcast for developers who love Postgres, where we discuss the human side of open source databases, Postgres, and the many Postgres extensions. A big thank you to the team at Microsoft for sponsoring this community conversation. I'm Claire Giordano.

PINO: And I'm Pino de Candia. Today's topic is From developer to PostgreSQL specialist.

CLAIRE: And I have the privilege of introducing our guest today, Derk van Veen. Derk started his career as a Java developer on a SaaS offering that hosted insurance data and over the course of the next five years Derk moved from being a developer to becoming a Postgres database specialist at Adyen.

Welcome, Derk.

DERK: Yeah, thank you very much. Honored to be here.

CLAIRE: We're glad you're here. All right, let's dive in. We have so many questions for you about why and how you made the transition from being a developer to becoming this expert on Postgres. I guess I want to start with what does it, what does it even mean to be a Postgres specialist?

Like what does your job at Adyen entail?

DERK: Yeah, that's a nice, that's a nice start. What does it entail? I would say best parts of my days are when things are not going as we planned, when the manager is stressed out and not everybody's working out as you planned something is slow and then you have to jump on it and fix it. I think these are my best days.

So yeah, basically jumping on problems, jumping on the problems, fixing long term problems. That's what I prefer to do. Spend a lot of time on tough problems and then find beautiful solutions.

CLAIRE: Okay. So that's the, almost the reactive part of your job, right? The detective work, the debugging, the solving, the fixing.

Is there also a proactive part of your day in terms of helping the rest of the developer team? Know what they need to know to try to avoid some of those problems

DERK: Yes, I think especially when it comes to partitioning, table partitioning within Postgres is really my topic. I automated it all and a lot of time, if I'm working with developers, I spent talking with them about partitioning.

How do you partition? Why do you partition? Why do we pick a certain strategy? Why do I think they should partition the table or should not partition the table? And the second part of interaction with developers is usually around performance.

PINO: And then do the developers usually come to you or, or do you often find you have to reach out to them to help them fix something that you think could be done better

DERK: When it comes to partitioning? They usually find me. And we have a system within Adyen every time a developer creates a table and this table is partitioned kind of alert goes off and they have to go to the DBAs and ask for approval. So even if they don't find us in advance, they will have to find us before they can actually commit this.

PINO: Oh, that's interesting. That's a great idea.

CLAIRE: So now I want to go back in time to 2010 when you were a Java developer. Was there, was there a moment, like a particular day or a particular month, something that happened that triggered you to change your path? From being a developer to becoming more of an expert on Postgres.

DERK: No, it was not not this bright moment or this, yeah, this moment where everything falls into place. I, I studied applied physics and the moment I graduated, I hardly knew what a database was my introduction into computers was already fairly late. So somewhere halfway around my study, I learned how to program and I, I really enjoyed programming.

And after I graduated, I got a software developer job at a company where I really learned. Yes, I do enjoy writing codes, but I don't like this company. So I already left them after half a year. I switched to the, to the company which makes software for insurance companies. And that's, that's where I basically got my introduction into databases.

And first, I was even a poor code writer at that moment. I had to learn how to write proper code. And only after a while you get the hang of it. And then I changed teams and I had to work for this one customer, which was at the moment, the biggest customer we had. I think it still is the biggest customer.

And if you are the biggest customer, you also have the biggest databases. And that was basically my introduction into databases that a customer is calling you like, Hey, I have a problem. My database is slow. How do I connect to the database? And how do I know what is slow? That's. I kind of got the hang of it and I got this magical feeling sometimes that if you're fixing something in the database, everybody's happy.

And I like doing this and I got better at it. And then I decided, Hey, we have a lot of databases internally as well. Maybe we should improve on this one on these databases as well. And then I became a manager of the internal DBA team. And after a while we transformed the company from a software company to a SaaS company and we had to do big databases for insurance companies on the SaaS platform.

And I had this smart moment where I said, I can manage internal databases, I can do SaaS as well. And with the knowledge I do have today about databases, I would not have said that. A, I did not know enough about databases by far to do this job. So I worked and I worked and I worked a lot for two years and then I was able to manage both DB2 and Oracle on a SaaS platform hosting insurance data for insurance companies in the Netherlands.

CLAIRE: Okay. So let's pause on that. You just revealed that you did not start on Postgres. You started on DB, DB2 and Oracle, is that right?

DERK: Yeah, actually, DB2. DB2 is my first database love, and I think I still, I would still enjoy working on it.

CLAIRE: What?

DERK: I'm sorry

CLAIRE: Can you say that On this podcast?

DERK: 'Yeah, I'm sorry. I haven't worked on it for five, six years, but I spent so much time on it and it was my first database, so I think I could still do it.

CLAIRE: I think it's really interesting that you just revealed too, that maybe you were a bit overconfident in the beginning. Saying that you could, you could manage these large scale databases in a SaaS platform and then finding out that, oh, I don't know as much as I think I did. That probably happens to all of us at some point.

There's something powerful to not knowing what you don't know. It gives you confidence.

PINO: Was it panic and learn or did you happen to have a gradual ramp up of those parts you didn't know?

DERK: I, first I spent a month on learning what I didn't know. And I was pretty scared, but we had about a year or a year and a half to build this entire platform.

So I basically had a year and a half to learn this and build it all. First I had to do Oracle. So I, I spent basically a year on Oracle, just reading everything I could find about Oracle, install it, work on it and and feel with it. I think these days I, I was working four days a week because my kids were smaller and I was taking one day a week.

I was taking care of them and I had a very smart colleague. And the day I was off, we had a deal, he broke the database. So when I arrived next day, I had to fix it. And we've played this game for a long time.

CLAIRE: He broke it on purpose to give you a challenge or he just did it without, without trying?

DERK: No, no, no.

He was actually breaking the database on purpose. So I had a tough job the next day. And this was a test database. There was some data in it and he was just, he was deleting files on the file system. He was screwing up my backup, deleting my backups. I remember this one day at such a dirty trick, he created a user on the file system Oracle with a zero width white space at the end of the name.

So when you copy the name, it's the same as the original user, but there were actually two Oracle users. But if you copy the name, you can't see the difference. And then he changed the ownership of half of the files to this new Oracle user, which appeared to be equal to the one I already had. It almost took me the entire day to fix this.

CLAIRE: Very devious.

PINO: Wow, that's trial by fire.

DERK: Yeah, but also, it was such a boost to my confidence, because by the time he had done this for months and months on me, I was really confident that, yeah, whatever happens, I can fix this.

PINO: You have to tell us how you took revenge.

DERK: I never did. I was actually very grateful.

PINO: No counter challenge.

DERK: I, I think I had my challenge when he joined the company. When he joined the company he never worked on Linux before. And I, I introduced him into the system and I said, well, if you want to change this, you have to change this file. He said, like my screen is only black with white letters. How can I change this file?

Oh, I said, we have this great tool on the, on the, on Linux, it's Vi and you open, you use Vi and then you can change whatever you want to this file. Oh, he says, how do I use Vi? Oh, it's easy. You take Vi- and then the file name and then you open it and then you're good. And at that moment I walked away

and yeah, he he made some weird noises behind his computer. But he he never asked how to fix it. But he did some Googling that day.

CLAIRE: Do you still work with this person?

DERK: No, unfortunately not. I tried to work with him again, but they made him a CTO of another company.

CLAIRE: Got it.

DERK: He doesn't work. I mean, he doesn't want to work with me anymore.

PINO: So I, I, I'm sorry. Go ahead.

CLAIRE: Well, I was gonna pivot to a different, a different question. So, Pino, if you've got something more on this thread, go for it.

PINO: Not so much this thread, but relate so, so something you said earlier, Derk caught my attention. So you said, your best days are when things are not going as planned.

You also talked about this magical feeling. And I know, at least for me, the magical feeling only comes once stuff is fixed and while stuff is broken, especially if people are relying on you to fix it quick. It's a hair on fire moment. So what is it like for you? Do you, do you do you stay calm?

You know, just explain the experience of things aren't going as planned. How do you approach that?

DERK: Yeah, usually first my heart rate doubles and I'm like, all right, this is not good. And usually I try first to take a step back. Like, just walk down the stairs, one floor, get a cup of tea, take a few deep breaths, go back up and then just start slowly and methodically on, on the issue.

Because if there's panic and your heart is pumping and you're immediately jump on it, at least that's how it worked for me. If I immediately jump, then I can't focus and I do the wrong things. So to my feeling, I really need to slow down on this, on these panicky moments. And only when I slow down, I can, I can analyze what's going on and try even try to relax.

Like this is, this is not a panic situation. This is just a moment to do a good job. And yes, the magical feeling is only afterwards when you have a beer and everybody sighs the breath of relief.

CLAIRE: I think that's really interesting. I mean, and I suspect that advice you just shared about needing to slow down, having a cup of tea, breathing.

Breathing right, so that your, your brain can focus is probably something that gets shared. Even, you know, outside of the developer space, outside of the database space, probably get shared by athletes, right? Probably get shared by surgeons, like, anybody who's got something very important to do. And needs their, their brain to be at peak performance.

Take a cup of tea. That's pretty cool.

DERK: It's hard if you're doing sports to take a cup of tea, but yeah, I try to be a professional dower and I came a long way. So I've been on some big tournaments. And yeah, before the match, it's like you're waiting in the water, you're, you can't move anymore. You're just waiting for your start position.

Yeah. And then it's just this breath, you know what to do, you know, the, you know, the skills, you have the skills, you practiced, just take a breath and then just do it.

CLAIRE: I love it. Okay. So you have worked obviously on Oracle and DB2 and now Postgres this podcast is aimed at the Postgres community, Postgres users, customers, developers contributors, community members. So I just have to ask, like, having worked with other databases and now having worked with Postgres for a number of years, what are the things about Postgres that have really appealed to you in comparison to those, those other, those other databases?

DERK: Before I started working on Postgres, I was like, if you hand so many tasks to the operating system, especially if you compare it with DB2 and Oracle, how can you get a decent performance out of a product? If you can't control all these variables, how can you do it? And then I started working with it, first at the insurance company, but it was introducing the database and it was like no load at all and I was getting bored.

But during this time I got involved with the Postgres community and I heard some rumors about a Dutch client with huge Postgres databases. And it was like, yeah, well, if they can do it, we can definitely do it. But yeah, I didn't know anything about it. But then I was bored and I applied for a job within Adyen.

That's where I'm working now. And indeed, we have huge databases and Postgres is just pulling it off most of the time. And for me, that's It's amazing. And sometimes I'm so surprised, like with relatively small amount of work, you can get so much work done at such a high speed. It's pretty cool.

CLAIRE: Are there, are there aspects of Postgres when you talk about being surprised? Are you talk about Postgres pulling it off? Like, are there pieces of the system components that surprise you in particular?

DERK: Not one in particular that I think of like, oh, it's amazing that that works, but there is so much of it.

If you have something and you want to do it, and it's like, you Google this, can I do this on Postgres? And usually the answer is, yeah, you can do this. And it works out. You can use JSON, Graph. I was looking into UUIDs last week, even this week. And you Google it and there's an extension for it. And then you can download the source code, compile it, work with it.

It's, it's just amazing that there's so much.

CLAIRE: So you mentioned extensions. I mean, that, that's a whole topic in and of itself. Because I started in the Postgres community working for Citus. So on the Citus extension I just became enamored with the power it gave the community. Like, and what I mean is if all new innovation, if all new capabilities had to be integrated into the Postgres core, then by definition, there's, there's some gatekeeping there, right?

And there'd be some limitations on, on the speed at which you could innovate or the types of things that could be done because you'd need to have sufficient agreement, etcetera, etcetera. I love the fact that the extension ecosystem. Is a bit more free. And, and allows for surprise, right? Allows someone to go create an extension because they believe in it.

Right. And then they can make it successful. Even when other people might've been like, Oh, I didn't realize that would be so useful. I don't know. Any thoughts on extensions? Am I alone in those, those feelings?

DERK: No, I think they're pretty cool and there's so many of them and they are, if they are proof they're worth, they, they get back to the core and you can even, even take it one one step further because people fork the Postgres projects.

And at first I was like, this is such a waste effort. There's so many people working on forks of the Postgres project and doing their own thing and then, but then it's, it's deviated from the core. And this weekend I was having a discussion with someone I met at FOSDEM, FOSDEM is a huge open source event in Brussels, which is in Europe.

And I, I talked to this to this guy I met there and I will say, I was saying this like, yeah, and then this forks and those effort is wasted. And then he said, no, you're looking at this from the wrong perspective. You fork it. So you can do all this development work. You can see whether it works out and you don't have to worry about breaking something because you're on your own fork.

And if it really works out and there's a market for it, a lot of people will, will use this or adapt to it, then the entire community will see the value and then it will be backported to Postgres core and then it becomes part of the core system. And then, yeah, maybe first it will become an extension and then you can use it.

And then in the end, the extension might go back to the core. And I thought this is a really nice way of looking at it.

CLAIRE: Yeah. Forks are part of the freedom of an open source project, right? For, for projects that have that, that kind of, you know, very liberal license.

DERK: Yeah, but at the beginning, I really thought these forks were a waste of effort.

Like, there are so many bright people working on such, on these topics, and then if you would have put all the effort into the core, then the core will develop faster. But now I see it more like, it's a nice way to, to get it done and to play with it, and then you can get it back to the core in a stable and safe way.

PINO: That's, that's, that's key, right? Certainly, the community goes slow in certain areas. Stability, performance are top top of mind and it's hard to get to get the community to, to to coalesce around something very ambitious. So it takes multiple cycles to do something. So I, I think what you're saying makes a lot of sense.

The, the, the, the forks help advance Postgres and and yeah, then maybe something can be done with a combination of, of, of development in core and, and and an extension when, when the idea is more mature.

DERK: Yeah, to me, it makes sense. At least it gives me a better feeling about looking to the system.

I think, yeah, everybody can, can contribute and there's no, nobody is in charge. So you can't make quick decisions overnight. Like if you are the boss of Oracle, you could say, Hey, this has to be done by tomorrow and then everybody jumps. But with open source, it doesn't work that way. So yeah, maybe you lose some efforts.

But on the other hand, you also can make very great, great products and great extensions to it and get it to the core and then develop even faster.

PINO: Now, on the, on the topic of extensions, I think we, we, we, I, for one, take for granted, you know, when I see another extension on my, the extension allows you to do so much that I sort of say, oh yeah, sure.

Postgres can do that because we have extensions, so I'm not surprised I can do this other thing and this other thing. And and people can do. So much just with the Postgres database are there, and so I meant to ask you, are the extensions that come to mind that you find particularly, you know, that you're particularly fond of that bring, that have brought a lot of value to you?

DERK: Yeah, basically for collecting statistics for us, our databases, we are really pushing it to the limits more than every now and then. So just being able to collect statistics, it's, it's key.

CLAIRE: So which extensions in particular are you thinking of?

DERK: Yeah, I was afraid you're going to ask this one, usually it's on top of top of my mind, but

CLAIRE: That's quite all right.

You'll think of it in a few minutes is once you stop trying to think of it. You know, one of the, one of the things we, we try to do in this podcast is. Give people specific suggestions and links. Like when we publish the, the podcast after the fact we will include show notes and there'll be a lot of links in there because we, we want to give people resources and tips and tools.

And what I'm curious about is you talked about reading everything you could find when you went on this journey from being a developer to becoming a Postgres specialist, do you, are there particular books? Or websites or resources that you can recall, and I know, I know I'm asking you to remember back to something you did five, six, seven years ago, but that you would recommend to someone who's new, who's trying to follow the same journey.

DERK: Yeah, it's hard, but what I think what you need as a, if you're starting with databases, it's just a general overview, what, what these creatures are, how do they breathe? How do they live? Because only if you know the basics, you know what you, what you're missing on. So I would say every generic database book, whatever the flavor database you use is, is already a good start because in general, all databases do the same thing.

They run up the operating system, they use memory, they have to do disk IO, they have execution plans, they have SQL, you need backups, high availability solutions, and you have to go through all of these topics to understand at least the basics of it and only when when you know what you need after that, then you can go to specific books and you'll learn who writes good blogs for your database flavor.

And for me, that was Ember Crooks, I think for, for DB2, I learned a lot from her blogs on she published them on Friday, Friday afternoon, usually. So for me, that's how I got used to it. Yeah. It's a general generic book, then a book about performance book about high availability setups. I think I even have Oracle book only about ARM backups.

And then you Google it and you find blogs and you find good blogs and then you keep on reading the blogs and that's how you learn. And even today, I can't, I can't work without Google or, or the manual.

CLAIRE: And when you say the manual, you're talking about the Postgres documentation or some other manual?

DERK: No, Postgres documentation, I know what I want and I know the general direction, but I don't know all the details.

PINO: And the Postgres docs are generally considered excellent.

DERK: I have to admit, yeah, they are good.

I have to admit.

CLAIRE: But you have to supplement them with Google because maybe you can't find everything you need in the docs, I'm guessing what, when you use Google to find information, where does it tend to point you to, to blogs?

DERK: I usually pick the documentation because the documentation is written by people who really know what they're writing about and when you're pushing a system to the limits, it's not about what you read, but what is actually there. So, sometimes you have to read it 4 or 5 times before you realize what is the picky, nitty detail.

That is still important and a blog more like I give you a general direction. It's a nice topic. This is how we use it. But if you want to be spot on, I think you, you have to use the documentation.

PINO: And Derk, I'm going to take a question from from the online chat. Do you use AI as part of your, of your work supplement Google and, and, and the Postgres docs? Or another other workflow.

DERK: Personally, I do not. I think our entire database team is not using it. I know the software developers are using it for some parts of their job to generate big parts of code, which can be generated. But for us no, we don't.

PINO: Oh okay. On the subject of extensions, I wanted to ask you too, do you, do you have to play a sort of defensive role?

Do the developers come to you asking to add extensions that they think might be useful, but it's hard for you to support another extension? Does that, does that happen?

DERK: Would say we are more on the careful side than developers, like a developer is building something and he's enthusiastic about it and he wants to get it live.

And when a developer comes to me and says, Hey, I need this extension. It's what I want to know is, is it safe? How does it impact performance? Is it well maintained or did nobody commit for this in years? So. I have more questions than just, yeah, on the surface, it looks good.

PINO: So it certainly takes a while to to onboard a new extension. I know that we have that, we have that issue at Microsoft as well.

DERK: Yeah, like, like this week, I've been researching this UUID extension, and it's a very small one. I think source code is not more than a few pages, but still, to get it approved, get it worked out, is the performance good enough?

Is it stable? Who's maintaining this? It easily takes months on the shortest.

PINO: And when you onboard it, does it happen the other way to that? You find an extension that you're pretty sure you're going on board and you need to let the, let the developers know, or even about the existing extensions, letting them know that the functionality is there, educating them that there is some functionality they may not, may not have taken advantage of yet.

DERK: Not directly, but indirectly we have we are collecting statistics about query plans, query execution times, these kinds of things. And we are published it using a a user interface and they are using it to take a look at their own database. Like, Hey, this is how your database is performing, but they don't know this information is provided by an extension.

PINO: Okay.

CLAIRE: So you talked earlier about partitioning and It sounds like Adyen does some pretty sophisticated things with partitioning, and I don't know what you're comfortable talking about or sharing here on the podcast or not, but I'm really curious if there are I don't know, is there a story there? Are there lessons you've learned that are worth sharing with the world about how you went from not using partitioning to now using it the way you do?

That's a very open ended question.

DERK: Yeah, well, the good thing is I can tell a lot about partitioning. I talked about it at conferences in the past. So people already know what kind of setups we have. We have articles about it and we actually published all of the functions we created to maintain partitioning or maintain partitions, I should say.

Because our system is always under pressure, and if you want to do maintenance on partitions under pressure, logging becomes a huge issue. And there were already some, some libraries out to maintain partitions, but we always struggled, like, yeah, the partition maintenance need this XS exclusive log, which is a very heavy log on the table, which basically means, everybody, don't touch it.

Don't even look at it. I'm doing maintenance. But if you're, if you're processing payments 24/7 a day, then you can't do this. So we basically created our own library to minimize the amount of logs we need when we are doing things like creating partitions, dropping partitions, but also even adding indexes to partitions or adding foreign keys to partition tables.

And, and partitioning is so easy to start with. Like, you basically just, you create a partition table and then that's it. And. I remember we did this on a Friday afternoon and I was like, I'm not sure what's going to happen, but I know we're going to hit problems. And yeah, of course that's what happened.

CLAIRE: So were you working that whole weekend then?

DERK: No, I didn't work over the weekend, but we made some mistakes of course. And in the beginning, we created the default partitions because the monitoring was not in place yet. And then suddenly you start writing information to the default partition and. It's like, Oh, that's, that was not the plan, but now I have a default partition and it's growing forever.

And how do I get rid of it again? And then you want to create indexes and creating indexes is not easy, or at least not that easy on partition tables. So it's not like working the entire weekend, but over time we had a lot of small problems.

PINO: Derk, you manage, you manage many databases, many different databases.

You, you're, you're, you, you talked a lot about a specific large and challenging database. But you manage several.

DERK: Yeah, at this point, I have to be a bit careful that working out of gen changes your perspective on big databases what some people would consider a large database. Sometimes we consider a medium size table.

So we have databases clusters, which are over hundreds of terabytes.

PINO: I see. So

DERK: We're still doing a huge amount of transactions on them so

PINO: What I was going to ask you with respect to partitioning is what are the parts of, of, of a partitioning design that you, you know, is it sort of standard you, you implement partitioning pretty much the same way in, in many of your database instances, or you have to redo the partition design every time, but, and you, but you can only bring principles along to help you do that.

DERK: I would say the most easy use case is we collect data. And when we. access the data. It's always based on a timestamp. So data comes in, it's nicely ordered by time. When it gets older, you can have a new, I don't know, a new partition for a new month. And in time it basically shifts around and all access patterns are also in time.

And then after time, maybe it's a log file you want to get rid of after a month or it's PII data and you want to get rid of it after five years. It's really easy. Everybody, everything is centered around time, but sometimes that's just not the way it goes. And then you have to be more creative because I think partitioning is already complicated.

If you have one layer of partitioning, so I don't want to go to two layers of partitioning and I don't want to have two partitioning columns. So we partition one column and one column only. And one layer. We don't do double layers, at least for now. And yeah, sometimes you're, you want to search your. Your dates or your data by a integer, which is your primary key and all access patterns are following the primary key or it's used for joints that it becomes much more complicated.

So how do you separate this in time? And yeah, then you have usually you have to make concessions both on database site where Give in a little bit of performance to find the correct partitions for your data, either at inserting time or retrieving time. And, and the code around it also becomes a bit more complicated.

Sometimes there's no, there's no way around it. And we have actually one use case where we use a huge default partition and we split this default partition in ranges. So we have within one huge partition, we have all kinds of sub ranges. And it also works out. But finding this solution, I think it took us months to find it.

So yeah, you have your, your, your guiding principles, you know, how it works, you know, your, your limitations and sometimes really easy and sometimes it's, it's really, it can be really complicated to find, to find something that really works out.

CLAIRE: You know, you just made that, that comment about usually you have to make some concessions. And it completely reminded me of something that Marco Slot said in his talk at PGConf EU about distributed Postgres database architectures. And he basically said that everything's a tradeoff. He said, usually to get something nice, you have to give up something nice.

I like the ring of that.

PINO: It's true, if you keep that in mind. Because we often don't think that, we often don't think that way. We think, we just have to think hard and we'll be able to figure out how to add something. And we don't really want to think about what we're losing.

DERK: I learned that us, you can't make an omelette without breaking some eggs.

CLAIRE: Yeah, that works too.

DERK: It's nice to have eggs and you can all, you can do all kinds of things with it. But if you want to have a nice omelet, you have, you have to give him something. And in my experience, that's how it works with databases. It's, it's not magic. It's not your way if you want, and then you can get everything you want to have.

You always, you give some and you take some.

CLAIRE: Yeah, it makes sense. Okay, so as a Postgres specialist at Adyen, the developers sometimes come to you to get input advice, guidance, and sometimes you go to them because you see something going on and you want to I don't know, I'm imagining you want them to make a change in their code or you want to advise them about a change you're making in your database.

The question is, that was all just the lead up, the question is, do you find yourself using analogies to explain database concepts to them, to explain some of these trade offs that have to be made? Like your egg analogy, but do you, and if you do use analogies, what are they?

DERK: Yeah. I was afraid of the second part of the question because I use them all the time.

It's usually, they just appear to me. I had a discussion with a developer a few weeks ago and I was just in a room and I, I noticed this, this stack of Post-its and then suddenly I started to use the entire table and Post-its and different colors of Post-its to, to explain different concepts when it comes to partitioning.

So it's just a Post-it and you write one letter on it and then you can already make it visual. I had the same thing about explaining rate arrays and I explained it to this person using what do you call these, these round circles you put under your beer glass?

CLAIRE: Coasters.

DERK: Yeah, these ones. And I use them to explain how a disc is rotating and how, how you can build a rate array and what are the trade offs there and how much can you lose.

It's more like using what I have around me and I use it to make visually. Clear what the concepts are. I think that's my trick. I would never meet a developer without a whiteboard in the room.

PINO: So it's not so much that you have standard analogies that you use, as much as whatever's available at the time, you're going to make use of it.

If it's visual, it's physical. It makes it more real.

DERK: Yeah, for me, that works out.

CLAIRE: Okay,

PINO: Definitely just was gonna add and definitely that's one of the challenges of remote work that you know, the whiteboarding isn't that great and and we can't really yeah. Yeah, we can't really draw on paper as easily we can't so so how do you get around that? What do you do in those situations?

You said you you said you didn't but does it ever come up if ever have a remote meeting and explain concepts. What do you do?

DERK: I usually I actually go to the office for them. For me, that's the reason to go actually to the office, to meet people, to, to talk people, go through problems, challenges, and meet people.

So you can actually do all these things, but right. Not all of the offices we have are in the Netherlands, so I can't always go to the office. So what do I do if I have to explain it. I'm pretty good with with keynotes to create slides and you can have very easy images on them. So I usually start with with keynote and then you make this very basic shapes and that's how we start.

I used to have this, this drawing tablet around me. I don't know where it is at the moment because it doesn't work with my Mac, I have now, but then it was more like you can have this page and then you can write with a pen already. It was already easier to work with than keynote. Yeah, you have to work with the limitations you have.

CLAIRE: Being able to write with a pen, I don't know why that is, how, like, biologically it's ingrained, that, that hand to brain connection exists, but I find it so much easier to sketch with a pen than I do trying to sketch, you know, with my mouse and online tools. So that's the limitation I have to work with.

So I asked you for examples of the analogies and you said you were afraid of that question because it comes to you at the time. Like you don't have some Rolodex of, okay, here's analogy number three that I'm going to share with you today. So that's fine. I get it. And I don't want to put you on the spot, but I do know of one analogy that you recently used that I thought was super creative and I'd love for you to tell everybody who's listening a bit more about it.

Like, how did you come up with the idea? And why does it work and how did people react to it? And the analogy has to do with the roller coasters, and specifically, this was the talk that you gave at PGConf.EU in Prague a couple of months ago and the title of the talk was Explaining PostgreSQL concurrency control mechanisms using roller coasters.

DERK: Yeah, these roller coasters were actually a very nice find and I tried different examples before that one first I started with the idea that you have a room and then you're not allowed to look at the room and then people leave or just come in and something's changed and when you look again, something has changed and because that's a bit like how transactions work within a Postgres database.

As long as you look at the database, it's, it's like nothing is moving. No data changes. But as soon as you blink your eye, suddenly everything has changed. But it's, again, it's quiet. Nothing's moving at that moment, as long as you keep looking. And for me, I started with this room with people. And if you blink your eyes, you're not looking for a while.

Then you look again, something changed. But as long as you keep looking, they don't move. And that analogy didn't work out in the presentation and in the story. It became way too complicated. So I had to look for another one. And then I was looking for an example where it should be moving at different paces because sometimes your database goes fast, sometimes your database goes slow.

But you also want to freeze it. And then I came, I was looking at what, what are the things around me that move fast and are kind of fun to watch. And that's how, how I finally found the roller coasters. And then I was explaining the concept to the team and how I wanted to connect it to to database transactions.

They were like, yeah, this is really cool. Especially when you have this rollercoaster and it's like this badge job is going to start and the rollercoaster is at the highest point and then all the speed is going. You know it's coming, but it's not there yet. And at that moment you can freeze the movie and have this picture of you can already feel it, right?

You can already feel how you're in front of this rollercoaster like, ah, it's coming. So yeah, that's how I found that analogy. And I think it worked out. It's, it was really fun to work on this, on this concept and play around with freezing images of rollercoasters.

CLAIRE: I love it. So this touches on something else, right?

PGConf.EU, and we already talked about FOSDEM PGDay. You were just there a couple of days ago in Brussels and you and Boriss Mejias jointly gave a talk together on high availability, if I remember and I know I first met you at FOSDEM last year. So, you give talks at Postgres conferences and you do this regularly.

As do I, and I'm just curious why you do it. Just a little bit of background too. I find this whole topic interesting. Like, why do people pour so much of their personal time and energy into preparing talks or organizing these community events? We had Boriss Mejías and Álvaro Herrera as guests on episode three of this podcast last year.

And, and actually that was the topic, why giving talks at Postgres conferences matters. So it's definitely like a whole conversation in and of itself, but short answer, why do you do it?

DERK: I think there are two answers to it. One is the more formal answer we are using open source software and I think it's a good habit to give something back to the community as well.

So at first I was convinced, like, what can I do? Well, I can program as well. I used to be a Java programmer, so maybe I can be a contributor to the Postgres code. And I tried it out and. I did not know where to start, to be honest. It's writing Java 10 years ago and now starting with a completely new project in a language I haven't seen in 20 years.

That was rough. And then I also realized that I'm good at other things. Within Adyen, we face a lot of tough problems and we have to do research about how to push Postgres to the next limit. And one of the things we can do is share this knowledge, we can share how we do partitioning, we can share what happens if you do this under pressure, and we can share the libraries we created to do this maintenance under pressure and share this knowledge, and yes, predicting the knowledge adds at events like FOSDEM and PGDays and PGConf.

I think that's also a way of contributing back to the community and say thank you. That's a more formal answer. But there must be a personal answer as well. Why do I do it? For me that answer is much harder to, to get I'm, I just like being on stage. While I'm usually pretty shy. That's when I'm on stage and I can share knowledge with people and I can see people like really understanding this thing you're trying to explain and they come to you after the talk is like, oh, now I understand why I need to do petitioning or now I know why it didn't work out or, oh, I'm so glad you guys already fixed it.

And now I don't have to do this. I can just follow it. That's what makes me feel personally good as well.

CLAIRE: I, I, I pause because I'm digesting what you're saying and it's pretty powerful. Yeah. There is something really cool to being on stage, and there is something exceptionally cool to sharing your learnings and your expertise in a way that helps people in their lives, in their jobs, with the problems that they're trying to tackle.

PINO: When was the moment when you thought, I'm ready to give a talk, I have something to share now, how much experience did you have? What was the trigger?

DERK: I think my first talk was for DB2 when I was just, we were, we had built a SaaS platform and I had collected a lot of information about performance debugging around DB2 queries.

And that was the moment I want, I said like, I, I've always been giving trainings. Like I was giving trainings internally to developers. I was training them when it came to databases, but also other topics. I remember being on high school and already my teacher asked like, Hey Derk, do you want to give this class today?

And I said, yeah, sure. I can give the class today. So it already started early, but really having my own stories to tell. Yeah, I think the DB2 one was the first one, and it was really bad. The slides were crappy. The story was crappy. It was way too short. I didn't know what I was talking about. At least I was, I knew what I was talking about, but I didn't know how to share it with an audience.

So they could actually follow what I was talking about. And then COVID came along and yeah, after COVID, it really took a good start again. And I was able to spend time on it, did a good training. We have a lot of really good people within Adyen and I did very scary things for myself because I built this presentation and this slide deck and I was proud of it.

I was really proud of it and I put a lot of effort in it. And then I went to the design team. We have some amazing designers at Adyen and I was showing my presentation and I said, what do you think of it? And he looks at it and he said, I think you have a good story, but your slides are really messy. And I think I spent days and days on these slides.

Well, I don't think so. I know so for sure. But then he explained me the basics about slide design and how do you make them more easy to digest for the audience and more calm and visually stronger. And then I learned a lot. So I approved on my slides and then, and then I went to sales and I said, how do you tell, how do you sell a story?

Because I'm a geek. I sell the truth. You guys, you sell a story. How do I, how do I sell a story to, to an audience? And, and then I learned from the sales team. Yeah, that's, that's how I, I learned and how I improved my, my skills. But still, if I'm in front of an audience, I'm still scared. Every day again.

CLAIRE: Are there any particular sales team tips on how do you sell a story that you can recall?

I know I'm putting you on the spot.

DERK: It always is about the why. Why do you share this? Why do you have to tell this to this audience? Why do you have this slide? Why is this visual on your slide? It's always about the why. Every part of it.

CLAIRE: So every part of what's on the slide, of what you say, needs to be intentional that you need to understand why it's there and make sure it's there for the right reason for your audience.

Is that what you're saying?

DERK: Yeah, because if you're, if you're working on a topic, you go left, you go, right, you do something stupid. Yeah, you make a mistake. You go back, you try it again. It's never a straight line, but if you're presenting the story to the audience, I think you should take the hand of your audience and guide them through the easy path.

So you tell them. What is in it for you today, before you start? And then, all the way through, you have to keep them on a straight line to the finish. There, there should not be any deviations, which should not be there. I learned that you have to kill your darlings, because you have been working on this side path, and the side path was so interesting, and you learned new things.

And it's good to share and it makes you feel like, ah, I'm so smart that I did this. And I want to share this to my audience as well. But I learned you have to, you have to cut the sidelines off. It's a straight path and you have to go there. And everything is, all the deviations are distracting for your audience.

And when you're on stage, you should not prove you're a smart person. You should share something your audience can use for me. That's my job. When I'm on stage, not showing off.

CLAIRE: Yeah, caring about your audience. Yeah, it's all about giving them a path that they can come with you on that. They're capable of like journeying with you.

That you have to kill your darlings quote. I just have to give attribution to that. That's Stephen King. And he obviously is a brilliant, brilliant fiction writer, but he also wrote a book of nonfiction that was part autobiographical about a really like important part of his life where he almost died.

He got hit by a car and you know, is lucky to be alive, but it's also a book about the craft of writing. It's called on writing and that's where I first got exposed to his famous quote about killing your darlings. And yeah, but maybe someone can drop the link to that book in the chat. If you want to become a better writer, that is like my number one writing recommendation book.

So how did you get exposed to the Stephen King quote? Where did you hear it? Do you remember?

DERK: I think my presentation coach told it to me, but I'm not completely sure on this one.

CLAIRE: Okay. All right. Well, I love, I love you sharing your learning from the sales teams. What about learning about slides, having them not be so messy?

Are there any actionable tips that you can remember from that? Those lessons?

DERK: Yes. But these lessons were the hardest one I ever had because when I'm talking to the design team, they say things like this slide doesn't breathe. And then I'm like, no, of course it doesn't breathe. It's a slide. No, no, no.

That's not what I mean. So you, you get this very weird discussions. So I, they learned me how to work with a design grid, which basically means every object you place on the same part of your slides every time over and over again. So it's always the same top line, the same bottom line, you always have the same space on the left and on the right.

This is how you make it stable. But for me, without having a grid to actually show it to people, for me. It's really hard to explain these concepts because they are so hard for me as well. I just, I just create my slides to my best of my ability and then I, I don't know it for slides. And then I go to a designer and say, this is what I'm trying.

Can you help me out? Because for me, it's, it's, it's really tough to do. I'm getting better at it, but it's not my nature.

CLAIRE: I'm, I'm going to share a tip and I'm curious if this ever came up in, in your learning journey with respect to slides. Larry Lessig used to present. Oh, I, I saw him present at conferences.

More than 20 years ago. And one of the things I realized with his slides is they were almost framing him. And by that, I mean that the focus was on him and what he was saying. And the slides were like the backdrop behind him supporting what he was saying. And, and so that's when I first realized that you had to kind of think about these slides as part of this holistic system that is the person.

What they're, you know, where are they standing? What are they doing? What they're saying? And then what's showing because that's what the audience is consuming. They're not consuming the slides by themselves. Right? It's, and, and so you don't want the slides to be so messy and distracting that they, they have to tune out the speaker because the visual is so complicated.

Does that, does that resonate at all?

DERK: Yeah, totally. The slide has to be easy and without you telling something with the slide, maybe the slide doesn't even make sense to your audience. It has to be very easy, easy to digest, easy to understand. The easier the better. It's a contribution to, to the story, to you as a presenter, to what you're trying to share with your audience.

If you want to share bullet points with your audience, just send them an email.

CLAIRE: I love that quote. We're going to write that down.

DERK: Yeah. People hate me for saying that.

PINO: That's very, that's very quotable.

DERK: Yeah. I'm afraid of that.

PINO: Well, actually, can I, can I ask Sorry, go ahead, Claire, if you were

CLAIRE: I just have one more thing I've been building up to that I have to say.

I wanted to give you a personal invitation, Derk. The POSETTE CFP. Oh, so let me back up. POSETTE is the new name for what was Citus Con: An Event for Postgres. So it's free and virtual organized by my team at Microsoft. It's called POSETTE: An Event for Postgres, and the CFP, the call for talk proposals is open until April 7th.

So I just wanted to make sure you knew about it and had the opportunity to submit talk proposals to that it's what I like about it is that these talks, because they're virtual, they are accessible to all the people who can't travel to these in person Postgres conferences, right? Because of travel budget or family or kids or whatever is going on in their lives, elderly parents, etc, etc.

And. I think it would be great if you submit a proposal. So there you go.

DERK: I will definitely submit more than one. I didn't want to do it before today. It sounds a bit weird, but I did want to catch up on some episodes I missed and I didn't want to sign up for the new event because I want to go into this podcast kind of fresh and crisp and not be influenced on things I might see or read or hear already.

So. Yeah. I will definitely submit and I know I have plenty of time.

CLAIRE: Yes. You have plenty of time. That's awesome. It's the, the event itself just for people listening to the podcast is happening in mid June. There'll be two live streams in America's time zone too and EMEA, sorry for the shameless plug.

I am head of the talk selection team. And so I'm very biased that I want lots of the amazing Postgres users and customers and developers in the world to submit because obviously, obviously that makes for, you know, a great event and lots of learnings to be shared. Okay. I'll get off my, my promotional soapbox here.

I think Pino had a question a second ago. Before

PINO: I did. I did. No, no, no. I'm glad you mentioned POSETTE. Claire, it's very exciting. The previous conferences have been really, really great. And and I'm, I'm just, I'm still looking forward to it. And I love it that it's virtual. I love that I can just that all the videos are there and yet there's also the community that's still live streaming so thank you for that.

I was gonna ask Derk, you know, we talked about about live presentations and I felt that just for completion, you know, just to ask you, you know, you've blogged too is it an equivalent? Is it an equal equal passion? Do you like doing it as as much or and what's different about it? Yeah. Just tell us about your experience blogging, please.

DERK: Blogging is nice because you can share more deep knowledge. You can really go to the details and explain it all. And in some ways it's easier to do because it's, you can take all the time you need. There's no, not like you have to fit in one A4 paper or it has to be readable within five minutes or 10 minutes.

So if you want to take, I don't know half an encyclopedia to share it, you can share it that way. What I'm always surprised about when I'm writing a blog is how much time it takes, and how much time it takes to get it right. And when it came to partitioning, we were like, oh, we will publish a new article every month.

And I think we are publishing a new article every three months. Because you have to do the research and you have to get it exactly right. And then I want to have some nice visuals as well. So I work together with the design team on the visuals, but.

PINO: So you have two articles on, on partitioning out late last year.

Is that right? Yeah. Is there,

DERK: is there a third part coming? We published it yesterday, I think, or the day before yesterday, so it's, it's really fresh out. And with the second block I created this public repository with all the functions to to do maintenance on the partitioning. So, yeah, then you have to build a repository and everything that comes with it, so that also took time.

But it's, yeah, it's, it's also a nice way of sharing and it's available for everybody for, for forever and ever. So it's not like you go to the conference and you can only share slides to a limited group of people. But on the other hand, going to a conference and talking to a lot of people, meeting people, gives me a huge energy boost as well.

So if I had to pick between going to talks with live audience and the blog, I would pick a few live audiences a year.

CLAIRE: I like having a mixture, Pino, because at the end of the day, I can reach more people with a blog. I can impact more people's day to day work, with a blog post. Just more people are gonna read it versus attending a talk. Although when you give an in person talk and you publish your slides and there's a video recording of it that then gets published on YouTube, that can really up the impact of an in person conference talk because you are then reaching more people than the ones in the room.

Then those privileged ones in the room.

DERK: Yeah. So our talk at FOSDEM was not recorded but me and Boriss want to have a recording anyway. So we will come together, have the slides recorded, and then we can still publish it.

CLAIRE: Yeah. And that's actually something you can do. You can self record and, you know, get it up there on YouTube.

And with all the tools these days, it's easier than ever to do that. And to have something that's high quality. Okay when, when Pino and I were talking to you Derk and preparing our episode brief, obviously, for those of you who are listening, we do not practice these conversations. Nothing is scripted.

Nothing is rehearsed. We're having this conversation on the podcast for the first time. However, we do have a set of 10 questions that we agree on up front. That's kind of like a safety net. Like if we're going down a path and we're not sure what to do, then, you know, I can always go back to that, that bag of 10 questions.

And one of those 10 questions, I'm just really curious about, I have this picture of you and Lætitia Avrot playing chess at the speaker dinner in Prague. Before PGConf.EU started and that's not the only time you play chess at Postgres events, I've heard. So, I'm curious, like, why do you play chess at Postgres events?

DERK: Well, if I can play chess, I just like playing chess. I'm a big fan of playing chess. It's this magical game with only 64 squares and half of them are covered by pieces. From a human perspective, the number of possibilities on the board is basically endless. And among the Postgres people, or more technical people in general, there are a lot of good chess players.

So I always carry my chessboard around and sometimes it brings excellent chess games and sometimes it has, has me meet new interesting people. I remember playing in Belgium last year where we were at a bar with maybe 20 people on one long table and then the chess board was on the top table and every 10 minutes everybody shifted.

So two new players on the chess board and you had new conversations. It was really fun.

CLAIRE: Oh my gosh, that sounds really fun. Is there something in particular that chess has in common with Postgres? I'm just fishing.

DERK: Yeah, it's problem solving. That's what I think. That's my job. My job is solving problems.

And that's what I really like to do. And chess is also about solving problems in time pressure.

PINO: I was going to ask you, do you use a clock on some or, or some of your games?

DERK: Yeah, usually we play with a clock on the conferences, it makes sure everybody gets a turn at the table. So you play for, I don't know, 5-10 minutes and then new players can can join the game or two new players can play and online, everything has a clock.

So I play daily chess where you have three days to play a single move. And I play games which should be finished within five minutes.

PINO: That's interesting. I think there's a little bit of an analogy you can make there where, you know, if, if, if you put the time limit too low, right, then you're just making mistakes all the time on both sides.

And the winner is the person who can make the fewest mistakes or can hold out the longest and not run out the clock, right? That there may be the Postgres analogy breaks down a little bit more, but do you like playing two minute, two minute games, one minute games?

DERK: No, I'm, I'm bad at the very fast games.

I'm better at the very long long games. The faster the time control, the lower my rating gets. And there's a huge gap between my fastest and slowest rating.

PINO: Absolutely.

DERK: But I would say playing daily chess is more like having a project with databases. You have a lot of time and you make slow progress.

While Blitz is more like you have to fix it right now and you have to survive. It doesn't have to be perfect, but just survive to the next day so you can work on something better then.

PINO: And you get used to getting, being sloppy. I've heard that, I've read that the, the, the Blitz games are not, it's not a good habit for serious chess players to play a lot of Blitz or speed chess.

DERK: No, it's more like I want to have something quick to do now, which is not work related. So I can just play a game for five minutes and then just get, yeah, I don't know, get rid of some energy.

CLAIRE: So I think you have an analogy for a future conference talk, somehow tying together chess with some of the Postgres performance problem solving that you have to do with intense time pressure. If you haven't given such a talk already, that is.

DERK: No, I never did. I never talked about performance. I submitted a few abstracts related to performance and partitioning, but never got accepted.

CLAIRE: Well, that could be because you submit so many good abstracts. And I say this as having been along with Andrew, who's listening in on the chat on the talk selection team for PGDay Chicago where I had to review a whole bunch of good talk submissions from you. Okay. We are almost running out of time.

Before we leave and I'll check with Pino to see if he's got any more questions. I'm just curious on this journey from being a developer to becoming a Postgres specialist. Are there any other tips or moments that you've thought of during the conversation today that I didn't give you a chance to talk about anything else you want to share with the world about that journey.

DERK: Well, I transitioned from a Java developer to a database specialist because I thought it was fun and it was appealing to me. And just because I think it is cool to do. And I think whatever you do, you have to follow your heart, whatever it is. Don't get stuck in some, some job you don't like and you're glad it's Friday.

Do something where your passion is. Then life and your job is just much easier.

CLAIRE: I, I, I call that the toothbrush test. When you're brushing your teeth in the morning, are you looking forward to what you're going to be doing at work that day? And my, my goal has always been to make sure I'm passing the toothbrush test in the mornings. And then I get in that the work I do causes me to get into flow states occasionally where I literally lose track of time, right?

Yeah. And so surprised, like what? It's eight o'clock in the evening. You're kidding me. I thought it was five o'clock. So yeah, those are the best days.

DERK: Yeah. I remember working on a, on a partitioning problem. We were working with, you can have a partition, but yeah, it's hard to explain in words. We have like ranges of partitions.

CLAIRE: You need a whiteboard.

DERK: Yeah. I need a whiteboard and then post it, but it was complicated question from one of the developers. Like, can you do this? I was like, no, well, well, maybe, and then I was working on it and I think I thought I was, yeah, I think I can make this work. And then I came home and I was like, so restless.

And I was like, yeah, now I know how it works. And then I started to do it. And I think I finished by two o'clock in the night, but then it all worked immediately. It's like, I want to have this done. I want, it's so cool. I'm so proud of it. I'm going to build it right now.

CLAIRE: Yeah, pretty cool. All right. Well, thank you so much for joining us. Really, really enjoyed this conversation today. Pino, before I wrap and go into the finale script, is there anything?

PINO: Nothing for me. I love it that you all ended on the toothbrush test and passion. That's why we're here. And so many of our guests bring up this, this exact topic.

So thank you.

CLAIRE: All right, well, thank you again for joining us Derk, and I look forward to seeing you again on the Postgres Conference Circuit. The next episode of this podcast will be recorded live on Wednesday, March 6th at 10AM PST.

The guests and topics are yet to be announced but you can mark your calendar now with aka.ms/PathToCitusCon-Ep13-cal. You can also get to past episodes and get links to the podcast on all the podcast platforms at aka.ms/PathToCitusCon, all one word. There's, there are transcripts included on the episode pages on Transistor too.

PINO: And wait, wait, wait, don't go before you go.

If you've been listening to this podcast, please rate and review us on your favorite podcast platform. It helps other folks find us and a big thank you to everyone who joined the recording live and participated in live text chat on discord.