| Auszug |
|---|
|
Nuclos performance large tables: chunk, chunkSize, avoid OFFSET, keyset, getEntityObjectsChunk, millions of records. |
Language: Deutsch · English
Reading and processing very large tables efficiently (millions of records)
Process large tables efficiently – read in chunks and replace OFFSET with keyset pagination.
| Status |
|---|
| colour | Green |
|---|
| title | Updated: Jul 2026 |
|---|
|
| Status |
|---|
| colour | Grey |
|---|
| title | applies to Nuclos 4.2026.x |
|---|
|
Two basic rules to process very large tables (millions of records) efficiently.
- Always fetch data in chunks.
- When reading the whole table, avoid OFFSET – page forward via the last ID instead (keyset pagination).
| Codeblock |
|---|
|
final long CHUNKSIZE = 5000;
long lastID = 0L;
Map<UID, FieldMeta<?>> mpFields = metaProvider.getAllEntityFieldsByEntity(entity);
DbQueryBuilder builder = dataBaseHelper.getDbAccess().getQueryBuilder();
for (;;) {
DbCondition cond = builder.plainCondition("t.INTID > " + lastID);
CollectableSearchCondition csc = new CollectableDbCondition(cond);
CollectableSearchExpression cse = new CollectableSearchExpression(csc);
Collection<EntityObjectVO<Long>> lstEOs = entityObjectFacade.getEntityObjectsChunk(entity, mpFields.values(), null, cse, 0L, CHUNKSIZE - 1);
if (lstEOs.isEmpty()) {
return;
}
for (EntityObjectVO<Long> eo : lstEOs) {
....
lastID = eo.getPrimaryKey();
}
} |
Related pages
Conventions for Nuclos API development
API.
Open →
