<p>If you have 4 parallel workers and need to process all rows from a SQL table, probably it's best to split the work with LIMIT/OFFSET into 4 parts and consume them with 4 tasks than to chose some smaller LIMIT/OFFSET and generate more than 4 tasks. OFFSET can get really slow and it doesn't build up speed when you make consecutive ones like OFFSET 1000, OFFSET 1100, OFFSET 1200, etc. In Python:
<code>limit = math.ceil(n_rows / n_workers)
offsets = range(0, n_rows, limit)</code></p>
By clicking “Accept cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts. Check our privacy policies.
By clicking “Accept all”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts. Check our privacy policies.