Skip to content
All lessons

Tech Insights

If you have 4 parallel workers and need to process all rows from a SQL table, probably it's best to split the work with LIMIT...

If you have 4 parallel workers and need to process all rows from a SQL table, probably it's best to split the work with LIMIT/OFFSET into 4 parts and consume them with 4 tasks than to chose some smaller LIMIT/OFFSET and generate more than 4 tasks. OFFSET can get really slow and it doesn't build up speed when you make consecutive ones like OFFSET 1000, OFFSET 1100, OFFSET 1200, etc. In Python: limit = math.ceil(n_rows / n_workers) offsets = range(0, n_rows, limit)

Shared by Flávio Juvenal

  • PostgreSQL
  • SQL

Book a call

Lessons like this one
shape how we build.

Tell us what you’re building. In one call, we’ll show you how our team would approach it.

Tell us what you’re building, and we’ll show you how we’d approach it.

Still prefer the good old email? Contact us