Large Data Volumes (LDV) in Salesforce is an imprecise, elastic term. These large data volumes(LDV) can lead to sluggish performance, including slower queries, slower search and list views, and slower sandbox refreshing.

Why is LDV important?

Under the hood

Salesforce Platform Structure

1. Metadata Table: describe custom objects and fields
2. Data Table: containing the physical records in your org
3. Virtual Database: A virtualization layer which combines the data and metadata to display the underlying records to the User
4. Platform takes the SOQL entered by the user which is converted to native SQL under the hood to perform necessary table joins and fetch the required information.

How does Search Work?

1. Record takes 15 minutes to be indexed after creation.
2. Salesforce searches the Index to find records.
3. Found rows are narrowed down using security predicates to form Result Set.
4. Once Result-Set reaches a certain size, additional records are discarded.
5. ResultSet is used to query the main database to fetch records.

Skinny Tables

What is a skinny table?

Condensed table combining a shortlist of standard and custom fields for faster DB record fetches.

Key Facts

1. Salesforce stores standard and custom field data in separate DB Tables
2. Skinny table combines standard and custom fields together
3. Data fetch speed is improved with more rows per fetch.
4. Do not contain soft-deleted records
5. Best for > 1M records
6. Immediately updated when master tables are updated
7. Usable on standard and custom objects
8. Created by Salesforce Support

Indexing Principles

What is an index?

• Sorted column, or combination of columns which uniquely identify rows of data.
• Index contains sorted column and pointers to the rows of data.

Standard vs Custom Index

Best Practices

How should we improve performance under Large Volumes?

1. Aim to use indexed fields in the WHERE clause of SOQL queries
2. Avoid using NULLS in queries as index cannot be used
3. Only use fields present in skinny table
4. Use query filters which can highlight < 10% of the data
5. Avoid using wildcards in queries, such as % as this prevents use of an index
6. Break complex query into simple singular queries to use indexes
7. Select ONLY required fields in SELECT statement

Ownership Skew

When you have more than 10,000 records for a single object owned by a single owner.

Why does this cause problems?

1. Share Table Calculations: When you move a user in the Role Hierarchy, sharing calculations need to take place on large volumes of data to grant and revoke access.
2. Moving users around the hierarchy, causes the sharing rules to be re-calculated for both the user in the hierarchy, and any users above this user in the role hierarchy.

How can we avoid this?

1. Data Migration: Work with client to divide records up across multiple real end-users
2. Integration User: Avoid making integration user the record owner
3. Leverage Lead and Case assignment rules
4. If unavoidable: assign records to a user is an isolated role at the top of the Role Hierarchy

Ownership Skew

When you have more than 10,000 records for a single object owned by a single owner.

Why does this cause problems?

1. Share Table Calculations: When you move a user in the Role Hierarchy, sharing calculations need to take place on large volumes of data to grant and revoke access.
2. Moving users around the hierarchy, causes the sharing rules to be re-calculated for both the user in the hierarchy, and any users above this user in the role hierarchy.

How can we avoid this?

1. Data Migration: Work with client to divide records up across multiple real end-users
2. Integration User: Avoid making integration user the record owner
3. Leverage Lead and Case assignment rules
4. If unavoidable: assign records to a user is an isolated role at the top of the Role Hierarchy

Sharing Considerations

Org Wide Defaults

What should I do

• Set OWDs to Public R/W and Public R/O where possible for non-confidential data
• Reduce the requirement for a share table.
• Use ‘Controlled by Parent’ to avoid additional share tables

Data Load Strategy

1. Approach- Middleware Led

2. Approach- Using Heroku?

3. Approach – Big Objects