PostgreSQL Development Conference 2024 - Review

Posted by ads on Friday, 2024-06-14
Posted in [Events][Postgresql][Postgresql-News][Vancouver]

This year’s PGCon was different. For starters, the conference is renamed to PGConf.dev, and it no longer takes place in Ottawa, but in Vancouver, BC. It’s also a new team which organized the conference, and they did a superb job!

Chelnik (left) and Slonik (right)
Chelnik (left) and Slonik (right)

Since its inception, PGCon was held in Ottawa at the University of Ottawa. I’ve attended it before. Now, PGConf.dev has moved to Simon Fraser University in Vancouver.

Simon Fraser University in Vancouver
Simon Fraser University in Vancouver

Conference Highlights

Tuesday started with the PostgreSQL Developer & Leadership Meeting, followed by either Advanced Patch Feedback Session, which was invite only, or alternatively the Extension Ecosystem Summit.

Wednesday and Thursday were reserved for the regular conference, featuring pre-selected talks, the keynote, and a reception for all participants.

Friday was unconference day, where everyone can submit a session, and then a scatter and gather takes place to merge all topics, and select the most promising ones.

And don’t forget the hallway track! Always good to talk to people.

PostgreSQL Developer & Leadership Meeting

The leadership meeting was invite-only, with 42 attendees. It started with reports from various PostgreSQL organizations:

  • Operations and Governance
  • Core Team
  • Security Team
  • PostgreSQL Community Association
  • Code of Conduct Committee
  • Port report

Some of this was a regular update from activities over the past year, some of this was new information. Most slides are uploaded on the PostgreSQL Wiki.

The group then split up into four different groups, and discussed several topics for an hour. Some of the topics were just discussed by one group, some topics by two different groups. After the break, the entire group gathered again and discussed the results. These are also found in the wiki.

There are not many decisions, or action items from this meeting. The only one is really that the group agreed that the project shall have a bugtracker.

Extension Ecosystem Summit

The afternoon was dedicated to the Extension Ecosystem Summit. David Wheeler prepared topic sheets, and attendees added their own topics as well. Discussions were held in two rounds to cover more topics.

Notes are in the wiki as well.

Personally I participated in the “Binary Distribution Format” session in the first round. We discussed how a binary distribution of extensions could look like, as well as related problems like installing multiple versions, or managing overlay builds.

The second session was about “Including/Excluding Extensions in Core”. This group discussed how it could be possible to include some of today’s extension functionality into core. This enables extensions which do not need any physical files on the server, and in turn allow everyone to distribute extensions without requiring superuser and filesystem access. Also a number of heavily used extensions like pg_stat_statements or pgstattuple can be directly in core, and not in contrib. These extensions do not require shared_preload_library changes, and having them in core will allow every user to use those.

Welcome

The regular conference began on Wednesday, opened by Melanie Plageman and Jonathan Katz, who both upped their ‘formal wear’ game each day. Everyone was invited to join, but I think not even Bruce was wearing a bow tie this time.

Anarchy in the Database - A Survey and Evaluation of Database Management System Extensibility

First session I attended was “Anarchy in the Database - A Survey and Evaluation of Database Management System Extensibility” by Abigale Kim.

She took an interesting approach by testing the extensibility of PostgreSQL when multiple extensions are loaded. This shows that many extensions are not compatible with each other, and loading too many extensions can lead to trouble. The talk was rounded up by showing stats from testing many extensions with other extensions, and how many do really work well.

Abigale Kim
Abigale Kim

Interesting TIL: PostgreSQL has 375+ extensions available, at least the ones which can be found in the public.

How Postgres is misused and abused in the wild

Karen Jex (Crunchy Data) shows in her talk “How Postgres is misused and abused in the wild” many horror stories how our beloved database is treated in the wild.

In a stark contrast the Otter Steve explains how developers can listen to their users.

Karen Jex
Karen Jex

The Postgres Ecosystem Foundation

Ad-hoc David Wheeler & Samay Sharma (both from Tembo) organized a meeting to discuss the potential establishment of a “Postgres Ecosystem Foundation”. 12 people from different companies participated in this discussion.

Raised questions include “did someone talk to core”, “what if we use another existing foundation” and “what if we do nothing”.

The conclusion in the end is to have a service like extensions.postgresql.org which acts as a registry for extensions, and other services can be build on top of it.

PostgreSQL 17 and beyond

Amit Kapila (Fujitsu) gave an outlook in his talk “PostgreSQL 17 and beyond”. First he showed some major features in v14, v15 und v16. Then he listed the new features in v17 in detail, and gave an outlook at v18 and more.

Some of the notable features in v17 include:

  • pg_basebackup has an --incremental backup option, the new tool pg_combinebackup reconstructs a backup
  • Several VACUUM improvements
  • Streaming APIs for faster reads
  • SLRU cache size optimizations
  • Improve performance of heavily-contended WAL writes
  • Sync/Failover slots for Logical Replication
  • Speed improvements for Logical Decoding
  • Allow partitions to be merged using ALTER TABLE ... MERGE PARTITIONS
  • Allow partitions to be split using ALTER TABLE ... SPLIT PARTITION
  • Allow exclusion constraints on the partitioned tables
  • MERGE command now supports RETURNING clause
  • MERGE command now supports WHEN NOT MATCHED BY SOURCE
  • MERGE command can modify updateatable views
  • Trigger on login events (even on standby)
  • Builtin collation provider
  • plenty of other SQL improvements
  • Allow certain VACUUM commands without superuser, using the new pg_maintain role
  • ALTER SYSTEM command improvements
  • New functions JSON(), JSON_SCALAR() and JSON_SERIALIZE()
  • New view pg_wait_events
  • Improved pg_stat_progress-vacuum views

Some of the interesting topics for v18 include:

  • Transparent Column Encryption (TDE)
  • Async I/O
  • Import/Export statistics
  • DDL Replication in Logical Replication
  • 64 bit XIDs

We will see how much of this will end up in v18.

Amit Kapila
Amit Kapila

Hayato Kuroda - Online Upgrade of replication clusters without downtime

Hayato Kuroda (Fujitsu) created the talk “Online Upgrade of replication clusters without downtimehttps://www.pgevents.ca/events/pgconfdev2024/schedule/session/20-online-upgrade-of-replication-clusters-without-downtime/" together with Vigneshwaran C (Fujitsu), and explains how pg_upgrade or Streaming Replication can be used to upgrade a PostgreSQL cluster from one major version to another one. He also shows the new improvements in v17 where the replication slot and subscriber information can be preserved during the upgrade.

Claire Giordano - How to Engineer Your Developer Blog Posts to Have Bigger Impact

Claire Giordano (Microsoft) has a very energetic talk about “How to Engineer Your Developer Blog Posts to Have Bigger Impact”

I’ve seen a previous version of this talk, and still did learn something new here. Sometimes I wish we all could have a reviewer for blog postings! Also: cute fluffy dogs!

Claire Giordano

Claire was also one of the volunteers who posted on the conference social media channels (Mastodon).

Making PostgreSQL Hacking More Inclusive

This was a very interesting session! The four speakers presented their viewpoints, and showed the audience examples how complicated it can be to contribute to PostgreSQL in any form.

Speakers (in order how they presented):

  • Robert Haas
  • Amit Langote
  • Masahiko Sawada
  • Melanie Plageman

My notes:

Robert:

  • We really really need new contributors (if we want to continue being awesome)
  • People get burned out, hacking on PG becomes less enjoyable because unfair

Amit:

  • His country (lives in Japan) prefers indirect, nuanced communication in personal interactions. However the PostgreSQL community uses a direct, assertive style prevalent in discussions
    • Maintaining two personas can be draining at times
    • Not being able to be direct and assertive may result in being overlooked or dismissed
  • There is also struggle in email conversations:
    • Difficulty expressing complex thoughts on email as non-native speaker
    • Time-consuming drafting process to sound coherent, potentially leading to perceived disengagement
    • Non-native speakers leading discussions may face reduced engagement
  • What can help:
    • Patience: Allow non-native speakers time to express themselves
    • Clear Language: Avoid jargon and idiomatic expressions for clarity
    • Clarifications: Request explanations or provide summaries when needed
  • Entrust newcomers with important tasks
  • Diversity Community Energy
    • Encouraging newcomers to tackle significant tasks not only broadens the pool of energy and ideas in our community but also boosts their morale
    • Also lays the groundwork for the development of specialists. Having a diverse group of specialists, each with deep expertise in specific areas, can enhance project outcomes.
  • Flipside
    • When newcomers aim for leadership roles in specific tasks, experiences members may hesitate, fearing it could impede project success, which may hinder recruitment of other experiences contributors.
    • Confidence in newcomers’s abilities has to be balanced with support and mentorship.
  • Off-list dynamics
    • Off-list collaboration can be very effective
    • However it may lead to some folks feeling excluded

Masahiko:

  • Being able to communicate in English is a prerequisite - he’s from Japan, not a native speaker
  • Most Japanese are not good at communicating in English
  • Require time to read and write in English
  • Communicate patiently
  • Misinterpret the “high-context” discussion in pgsql-hackers
  • Experience of emails not being replied to or failing to get attention
  • Delivering negative feedback
    • Direct and indirect communication
  • Time zone differences
    • 140 emails on pgsql-hackers per day
    • Most emails at night time in Asia
    • Less opportunity to engage in the discussion

Melanie:

  • Listen to women who want to stay technical
    • People become good at things they do
    • All technical leaders on resilient teams have social responsibilities
    • ICs (Individual Contributor) should strive to be great mentors too
    • Technical work should be the primary ask
  • Invest in Developers at every level
    • Explain from basic principles on the “hallway track”
    • Teaching a workshop or educational meetup is a statement of our priorities
  • Communication Efficiency Goals
    • Avoid ambiguity (leads to harsh discussions)
    • Spend time on the right discussion
  • Responding on the Mailing List
    • Be specific
    • Speak from your experience
      • This sounds like a completely terrible idea => I don’t think this is going to work
    • Reply to the person (not just the content)
      • (Dialect of English on the mailing list)
    • Don’t talk around people
    • Highlight the positive

Robert again:

  • Conferences
    • You can bring family members to a PostgreSQL conferences
    • Conference social events tend to revolve around alcohol consumption
    • When someone attends a conference for the first time, how much attention do they receive from the other attendees?
  • Working Hours
    • If you break buildfarm, or CI, you are expected to fix it
    • Your social life, family life, and mental health will have to take a back seat to the community obligations
  • Respect
    • Check your bias
    • Stay positive
    • Be mindful of your own emotions

This was then followed by a Q&A session:

  • “Has the Postgres Foundation ever considered hiring a manager which can triage the work?”

    • Robert: “Was considered on Tuesday”

  • “Is there a way to accelerate the way to get things done (patches)”

    • Robert: “Would be nice, is also very difficult”

  • “What is IC”

    • Melanie: “Individual Contributor”

  • “Problems with understanding people with different dialects, is it helpful to put text on the slides because it helps understanding the context”

    • Masahiko: Thank you

  • “Indirect form of communication in Japan, how does that work with code review”

    • Masahiko: “Come up with another, indirect phrase: this code is terrible, is there another option to go”
    • Robert: “The four people (here on stage) couldn’t agree on how to say no, which shows how hard this is”

  • “How did Robert encourage you to work on partitioned tables”

    • Amit: “Off-list dynamics, Robert was interested, found that I’m willing to spend time”
    • Melanie: “I asked Robert if I can have a monthly meeting, if I have an agenda.”

from left to right: Robert Haas, Amit Langote, Masahiko Sawada, Melanie Plageman
from left to right: Robert Haas, Amit Langote, Masahiko Sawada, Melanie Plageman

All in all this was very good insight. I had training on cultural differences before, and I’m always open to learn about different viewpoints. In my view it was good that this was scheduled as a main session for everyone, and not just another talk with parallel sessions going on. This way, everyone had a chance to attend this session.

Lightning Talks

Lightning Talks are always fun, and a good way to quickly learn about a new tool or feature. This was no difference here.

Improving extensions in core

This was one of the two Unconference sessions I attended. My notes:

  • API/ABI stability
    • Core is focused on core ABI stability
    • David: No “statement of stability” in Core
      • David/Jeremy/Tom: coding guidelines, style guidelines
      • useful to have docs in core about what’s stable and what’s not, what you should compile against or not, and ABI guarantees
    • Abigale: there are hooks, but no overall concept for extensions
      • Tom: Peter E. is working on tests for extensions stability
      • Jeremy: nothing is preventing people from installing incompatible versions
  • Abigale: extensions manager needed, in Core
    • More smoke tests needed
    • Grant: extension update during major PG version upgrades is a major problem
      • Chris: does this require a compiler on the host when upgrading extensions
      • Jim: how do we solve problems with problematic extensions, like when they be need to be loaded in specific orders
      • Abigaile: there are tools to solve this problems with tests
      • Andreas (to Jim): you only need a dependency graph for problematic extensions
    • Abigaile: I want to categorize the errors
    • Kaiting: Are we also talking about a package repository
      • Abigaile: yes, ideally
      • Andreas: a central repository (for packages) will also improve security, by providing a safe central place
    • Kaiting: how do you maintain the dependency graph
      • David: manually
      • Chris: is it source based, or binary based
        • David: yes
    • Paul: Instead of having hooks, having rails to call hooks from multiple extensions
    • Chapman: What’s left to be solved which PGXN does not solve
      • Abigaile: couldn’t work with PGXN, doesn’t work, using multiple extensions in PGXN doesn’t work
      • Abigaile: currently it’s not possible to test multiple extensions
      • Abigaile: have a description of what the extensions is modifying
      • PGXN v2 can solve these problem
    • Hannu: we can only do extensions matrix if we also cover extension testing
    • Jeremy: like that Abigaile started to categorize the extensions, and what exactly they are extending
      • That’s right now it manual effort
      • Jim: instead of extension manager picking the end user, build it for the DBA, use PG.org
        • Works better, scales better, allows Devrim to build better packages
        • Chris: it solves the problem of solving other operating systems
        • Yurii: we can look at the .so files and use this for categorizations
        • Abigaile: wrote a script, listed all the hooks, kept track of the hooks and SQL functions to categorize
          • Can be improved, is regex-ing on SQL files right now
          • Being in Core has a lot of power which is right now not there
    • Kaiting: central repository, use a fork and use an extension to categorize
      • Yurii: this can be done with an extension
    • Yurii/Abigaile: Coalition
      • Group to be created to manage extension efforts
        • Write documentation how to write extensions
        • fork of PostgreSQL, if there is something not working
      • Yurii: the fork is there to test changes in extensions
      • David: there is a GitHub organization, maybe a mailing list
        • Andreas: how about the Slack channel
          • David: it has grown, but it feels like I am talking and others are listening
    • David: inline extensions
      • There was a patch in 2014 which provides inline extensions, without files
    • Hannu: Aliases
      • If parsing fails, instead of failing out, pass it to another parser
    • Hannu: other languages
      • Jim: pl/Rust is complicated, because it needs crates and compiler
      • Jim: TLE allows WASM without file access (whole package, including control file)
    • David Wagoner: immutability support
      • Christoph B. had a patch
      • Second dir allows immutable images, and a second dir with data and extensions
      • Hannu: having a virtual file system (Fuse) allows reading from the database
        • David: sounds complicated

All in all a very productive meeting. I’ve seen more discussions around these ideas in the days after the conference.

Table AM API

The second session was presented by Andrey Borodin (Yandex) who was on-site, and Alexander Korotkov (OrioleDB) who was connected over video.

The proposal shown in the presentation is to extend the Table AM API. My notes:

  • Alexander: currently the row must be identified by ctid (32 bits block number, 16 bits offset number)
    • Some code path assumes that row versions are identified by ctid
    • Introduce RowRefType and TableAmRoutine
    • Allows ByteA row identifier
      • Some indexes can support that
      • Bitmap scans should be disabled, or replaced
    • Use cases:
      • Index/organized tables
      • LSM indexes
  • Hannu: why can’t you just use the existing methods, keep ctid as it is, and add your identifier on top
    • Matthias: you need this for indexes
  • Jeff: it the proposal also to support update chains in read-commited level
    • yes, planned
  • Matthias: does it imply that you also work on a table AM that does not use block storage
    • Alexander: currently we have pluggable storage manager
  • Hannu: it’s already a lot of changes, why are you trying to change the table am, and not try to change the foreign table wrapper
    • Alexander: don’t have an answer
    • Jeff: can’t have an index on it
  • Andres: concern is that this is a massive change, without a lot of benefits for most users, unlikely to succeed, stability issues, needs more benefits
    • Chris: what is the story if not pluggable types are supported
    • Andres: you also need to support PostgreSQL
    • Heikki: can index-organized be a first step
    • Matthias: don’t start here, start with the APIs, this can support index-organized tables, and also removes Vacuum for index
    • Alexander: avoid a large patch, which changes everything
      • We will update this
    • Jelte: making a path for index-ordered tables is more useful for many people
  • Hannu: why are you using table am, not index am
  • Chris: we need an ability to tie certain planner nodes to certain table am, in order to get column based tables
    • Jeff: should we have more than one relfilenode per table
    • Chris: you almost want something which is vectorized, once you get into different table am types for different access patterns when it’s no longer a heap table

It felt like this is something which is already implemented, and now looking for consent from the community. The resulting discussion in the room shows that this raises more questions than answers. There was no clear decision in the end, but several more discussions spun off in the hallway track and after the session.

Increase community participation

This was the other very interesting Unconference session which revolves around how to increase the community, include more people, and ensure that people in the community do not burn out. It was organized by Stacey Haysler and Joe Conway. My notes:

  • Not everyone knows what to do
  • Make it easy for people to volunteer
    • PgUS has a form on the website
  • Rob: we are in a “don’t ask” community, I’ve been asked and then nothing happened
    • This conference needs more people, call for volunteers
    • A lot of the other things also need to happen, and it’s not obvious
  • Chris: people are not aware of the things outside of the code
    • It"s relatively easy to find support as developer (Robert disagrees)
  • Joe did a presentation at Seapug
    • Grant: is any of this documented?
  • Stacey: how do we support the people
    • Connect to the people
    • Conferences
      • Organizers
      • Selection Committee
      • Volunteers
      • Sponsors
      • Speakers
      • Attendees
    • Chris: extend the Wiki to include people
    • Chris: make sure people see value in support
    • Claire: making people more visible is important, make it easier to navigate
    • Claire: recognise the people who contribute outside of code
    • Grant: we think about the things core to the technology
      • What’s missing how to engage the end user
    • Stacey: think about continuity
    • Kaiting: people are understaffed
      • It needs enough people to reach that activation energy, to make peole’s life less suck
    • Angie: sit down and define roles for people
    • Stacey: average time for tasks is not known
    • Stacey: DjangoCon has 35 people in the planning, twice as much volunteers
    • Robert: we are not doing a good job helping people to become successful in terms of becoming developers
      • Stas: imagine showing up to a room full of people, try to figure it out
    • Robert: we need a way to give people more support
    • Robert: we are not going to have more developers the way it’s currently done
    • Chris: successful contributors have a mentor in the community
    • Jonathan: extension developers are a big part why PostgreSQL is successfull
      • have an extension summit
    • Jonathan: also drivers, and tools
  • Stacey: How do we reward people
    • PgUS has coins
    • Small number of non-code contributors in PostgreSQL get a coin
    • Major contributors are not announced
    • Joe: Contributors Committee
    • Vik: add a link on each committee page: “volunteer here”
    • Jonathan: it’s important to have fun with it
    • Robert: wouldn’t it be cool if we have a feed for it, awesome stuff someone did
      • Andreas: show this feed on the pg.org website
    • Grant: also have sections where committees and groups need help now, compared to general help
      • Like for conferences and such
      • And have good descriptions what is asked for
      • Help needed on the website
    • Ilya: there are several newsletters, and they struggle for content
      • Add an ordinary blog which summarizes contributions
      • “5 minutes in Postgres” video? blog?
    • Chris: there was once a todo list
    • Noah: make sure that any dashboard is checking different things, not just mailing lists
  • Stacey: to newcomers
    • Does anything of this makes sense to you?
    • James: I’m hesitant how contribute to long-time commitments
      • Little things are helpful as well for volunteers
        • How to get the info to these people?
    • Angie: why do people stay
      • People got 1:1 support (mentoring)
      • A lot happens outside of mailing lists, mostly in person
      • Joe: how, we don’t want to impose mentors on people on people who don’t want that
        • Robert: I have an idea about that
        • Robert: mentoring can be extremely rewarding
        • Andreas: have a community manager who peers newcomers with mentors
        • Claire: was part of a mentoring ring, in a small group you have fruitful conversations

My takeaways:

  • A dedicated “app” conference, possibly with other projects like Django can bring more people to the project
  • The project needs a better way to recognize people, especially people who do not or not only contribute code
    • In addition improve the main website and show where people can contribute
  • Mentoring for new developers (and potentially other areas as well) can help overcoming the initial grind to get a patch accepted

Joe Conway (left) and Stacey Haysler (right)
Joe Conway (left) and Stacey Haysler (right)

More Unconference

The notes from the unconference are available in the wiki as well. Unfortunately, I missed the multithreading session, but it seems the project is making progress towards switching from processes to threads.

End of Conference

And then it was over, and time to close the conference. Melanie and Jonathan thanked everyone for coming to Vancouver, and reminded everyone to submit feedback.

Melanie Plageman (left) and Jonathan Katz (right)
Melanie Plageman (left) and Jonathan Katz (right)

Other activities

There have been a couple of other activites at and around the conference, including:

And of course there were stickers!

Stickers
Stickers

Other blog postings and videos

Categories: [Events] [Postgresql] [Postgresql-News] [Vancouver]
Tags: [British-Columbia] [Canada] [Chelnik] [Comment-Option] [Conference] [Database] [Django] [Event] [Extension] [Interview] [Kanada] [Konferenz] [Lightning-Talks] [Ottawa] [Pgcon] [Pgconf.dev] [Pgconfdev] [Postgresql] [Presentation] [Slonik] [Talk] [Unconference] [Vancouver]

Comments

With an account on the Fediverse or Mastodon, you can respond to this post. Known replies are displayed below:



Note: This will load data from mastodon.social.

Learn how this is implemented here.

Share:

Andreas 'ads' Scherbaum

Friends