Monitor this situation.
Unsubscribe anytime.
[SITUATION] · [ACTIVE] · [TECHNOLOGY]
3 clusters · 6 sources · 25 days · First seen · Last updated
PostgreSQL data management and migration strategies
Overview
Technical discussions regarding PostgreSQL focus on various strategies for data migration, scaling, and security. Methods for moving data include using native logical replication to migrate specific tables when automated services are limited to entire databases, and utilizing offline migration approaches to manage capacity limits such as disk space or CPU constraints. To optimize initial data dumps and avoid constraint errors, practitioners may temporarily create tables without indexes or foreign key constraints.
Scaling and security techniques are also addressed. To manage large-scale data, practitioners may transition from PostgreSQL to Hadoop, though it is noted that such specialist solutions can introduce complexity and maintenance costs. For security, developers utilize role-based access control (RBAC) and specific schema designs to implement row-level security for views, ensuring users only access designated data. In multi-region setups driven by regulatory needs, sharding is used to move tenant data across instances.
Recent implementations highlight advanced architectures for large-scale environments. Lakebase Postgres utilizes a tiered architecture with a local compute cache to mitigate latency issues associated with object storage, ensuring object storage is accessed in only approximately 1 out of every 28 million requests. Additionally, Zerodha manages massive instances on bare EC2 instances, utilizing a sharded architecture of four nodes to store nearly 20 TB of data. To handle peak traffic, they employ a specialized setup where one PostgreSQL instance acts as a hot caching layer, supporting up to 50,000 queries per second.
Entities
PostgreSQL · Google · PostgreSQL · Lakebase · Hadoop
Timeline
-
[TECHNOLOGY] 2 sourcesPostgreSQL performance and management in large-scale environments
Technical insights detail how Lakebase Postgres uses tiered caching to optimize object storage reads and how Zerodha manages 20 TB of sharded PostgreSQL data on EC2.
-
[TECHNOLOGY] 2 sourcesPostgreSQL data management and security techniques
Technical insights cover implementing row-level security for PostgreSQL views and strategies for migrating tenant data across database instances during sharding.
-
[TECHNOLOGY] 3 sourcesPostgreSQL data management and migration strategies
Technical insights cover the strategic transition from PostgreSQL to Hadoop for big data needs and the use of logical replication for granular table migrations within PostgreSQL instances.
Sources
databricks.com · immunology.utoronto.ca · martinbelam.com · our.umbraco.com · via.ritzau.dk · zerodha.tech
This summary has been updated 1 time: see revision history