Skip to main content

DHTMLX Pivot. Large dataset example

Demo of loading large datasets into DHTMLX Pivot with performance benchmarking. Switch between 50K, 100K, 500K, and 1M rows to test data generation and rendering speed in real time.

Live example​

const config = {
    rows: ["hobbies"],
    columns: ["activities", "language"],
    values: [ 
        { field: "age", method: "min" },
        { field: "age", method: "max" },
        { field: "height", method: "max" }, 
        { field: "weight", method: "max" },
    ],
};

const select = document.querySelector("#form-select");
const labels = ["50K", "100K", "500K", "1M"];
const dataSizes = [50000, 100000, 500000, 1000000];

let size;
let gStart;
let gEnd;

// generate options for the select control
dataSizes.forEach((id, i) => {
    const opt = document.createElement("option");
    opt.value = id;
    opt.innerHTML = labels[i];
    select.appendChild(opt);
});

// generate data and set it to Pivot
function loadData(index) {
    select.value = size = dataSizes[index];
    const data = generateData(size);
    init(data);
}

function updateStats({ rStart, rEnd }) {
    document.getElementById("log-area").innerHTML = `Time to generate data: ${(gEnd - gStart)/1000} seconds <br> Time to render data in Pivot: ${(rEnd-rStart)/1000} seconds`;
} 

function init(data){
    const rStart = new Date();
    pivotWidget.setConfig({ data });

    // measure the (approximate) time it takes to render new data
    requestAnimationFrame(() => {
        setTimeout(() => {
            const rEnd = new Date();
            updateStats({ rStart, rEnd });
        }, 1);
    });
}

const pivotWidget = new pivot.Pivot("#pivot", {
    fields, 
    data: [],
    config,
});

loadData(1);

<!-- auxiliary controls for interacting with the sample -->
<link rel="stylesheet" href="https://snippet.dhtmlx.com/codebase/assets/css/auxiliary_controls.css">

<div style="display: flex;flex-direction: column; width: 100%; height: 100%;">
    <div class="dhx_sample-controls"style="display:flex;flex-direction: column;background: #fff;height:150px;">
        <div id="log-area" style="height:40px;"></div>
    	<div class="select" style="height:60px;">
            <label for="form-select" class="dhx_sample-label">Select dataset size:</label>
	        <select name="form-select" class="dhx_select dhx_sample-input" id="form-select" style="width: 220px;" onchange="loadData(this.selectedIndex)"></select>
        </div>
    </div>

    <!-- component container -->
   <div id="pivot" style="height: calc(100% - 150px); width: 100%;" />
</div>

<script>
// data generator
    function generateData(size) {
        const data = [];
        gStart = new Date();

        for (let i = 0; i < size; i++) {
            data.push({
                name: getRandom(names),
                age: getRandomNumber(18, 100).toString(),
                height: getRandomNumber(150, 200).toString(),
                weight: getRandomNumber(50, 150).toString(),
                date_of_registration: getRandomPastDate(),
                hobbies: getRandom(hobbies),
                favorite_color: getRandom(colors),
                favorite_pet: getRandom(pets),
                relationship_status: getRandom(relStatuses),
                favorite_book: getRandom(books),
                music: getRandom(musicGenres),
                favorite_video_game: getRandom(games),
                education_level: getRandom(education),
                language: getRandom(lang),
                favorite_dish: getRandom(dishes),
                favorite_drink: getRandom(drinks),
                favorite_dessert: getRandom(desserts),
                activities: getRandom(activities),
                activity_level: getRandom(actLevels),
                fashion_style: getRandom(styles),
                places_to_visit: getRandom(places),
                movie_preferences: getRandom(movieGenres),
                personality_type: getRandom(personalityTypes),
                superhero_power: getRandom(powers),
            });
        } 

        // measure the (approximate) time it takes to generate new data
        gEnd = new Date();

        return data;
    }

    function getRandom(list) {
        return list[Math.floor((Math.random() * list.length))];
    }

    function getRandomNumber(min, max) {
        return Math.floor(Math.random() * (max - min + 1)) + min;
    }

    function getRandomPastDate() {
        const minDate = new Date(2010, 2, 3);
        const currentDate = new Date();
        currentDate.setHours(0, 0, 0, 0);
        const minTime = minDate.getTime();
        const maxTime = currentDate.getTime();
        const randomTime = Math.floor(Math.random() * (maxTime - minTime + 1)) + minTime;
        const randomDate = new Date(randomTime);
        randomDate.setHours(0, 0, 0, 0);
        return randomDate;
    }

    const activities = ["Hiking","Swimming","Cycling","Running","Yoga","Painting","Cooking","Gardening","Reading","Writing","Photography","Playing musical instruments","Dancing","Playing board games","Watching movies","Birdwatching","Fishing","Camping","Traveling","Volunteering","Meditation","Knitting or crocheting","Sculpting","Woodworking","Pottery","Playing video games","Rock climbing","Surfing","Skiing or snowboarding","Playing tennis or other sports"];
    const names = ["Liam","Olivia","Noah","Emma","Oliver","Ava","Elijah","Charlotte","William","Sophia","James","Amelia","Benjamin","Isabella","Lucas","Mia","Henry","Evelyn","Alexander","Harper","Mason","Abigail","Ethan","Emily","Michael","Ella","Daniel","Madison","Logan","Avery","Jackson","Scarlett","Sebastian","Lily","Jack","Grace","Gabriel","Chloe","Carter","Ellie","Matthew","Layla","Jayden","Nora","Luke","Mila","David","Aria","Wyatt","Hannah","Joseph","Elizabeth","Owen","Penelope","Dylan","Victoria","Samuel","Luna","John","Sofia","Anthony","Eleanor","Isaac","Addison","Adam","Aubrey","Nathan","Camila","Ryan","Stella","Evan","Zoe","Thomas","Hazel","Christopher","Natalie","Joshua","Leah","Andrew","Audrey","Christian","Violet","Grayson","Claire","Lincoln","Bella","Isaiah","Lucy","Aiden","Aria","Eli","Nova","Hudson","Emilia","Mateo","Aurora","Leo","Luna","Ezra","Ariana","Asher","Elena","Grayson","Naomi","Jacob","Paisley","Levi","Willow","Silas","Ivy","Ezekiel","Hazel","Finn","Gianna","Theodore","Isla","Nathan","Savannah","Axel","Eva","Miles","Ruby","Jameson","Valentina","Luke","Everly","Jayce","Clara","Carson","Eleanor","Sawyer","Valerie","Jason","Isabel","Xavier","Audrey","Colton","Skylar","Gabriel","Jasmine","Jaxon","Adeline","Maverick","Josephine","Kingston","Alyssa","Caleb","Eliana","Weston","Summer","Thomas","Delilah","Isaiah","Juliana","Hunter","Amara","Julian","Leilani","Lincoln","Alexa","Aaron","Lila","Landon","Alana","Everett","Serena","Nathaniel","Genevieve","Roman","Gabriella","Elias","Brielle","Evan","Angelina","Bryson","Daisy"];
    const pets = ["Dog","Cat","Hamster","Rabbit","Parrot","Guinea pig","Fish","Bird","Turtle","Horse"];
    const colors = ["Red","Blue","Green","Yellow","Purple","Orange","Pink","Brown","Black","White"];
    const hobbies = ["Photography","Traveling","Reading","Sports","Cooking","Hiking","Gardening","Playing musical instruments","Writing poetry","Knitting or crocheting","Birdwatching","Yoga or meditation","DIY crafts","Playing board games","Collecting stamps or coins","Learning a new language","Fishing","Sculpting","Sewing or quilting","Playing video games","Volunteering","Model building (e.g., airplanes, cars)","Astronomy","Home brewing or wine making","Geocaching","Archery","Beekeeping","Calligraphy","Pottery","Origami","Horseback riding","Martial arts","Interior decorating","Camping","Surfing"];
    const books = ["Harry Potter series","The Lord of the Rings","The Alchemist","The Da Vinci Code","The Twilight Saga","Gone with the Wind","The Chronicles of Narnia","To Kill a Mockingbird","The Catcher in the Rye","The Hunger Games","The Fault in Our Stars","Fifty Shades of Grey","The Great Gatsby","1984","Pride and Prejudice","The Hobbit","A Tale of Two Cities","The Kite Runner","The Lion, the Witch and the Wardrobe","The Little Prince","The Girl with the Dragon Tattoo","Catch-22","The Help","The Giver","The Secret Garden","The Adventures of Huckleberry Finn","The Outsiders","The Bell Jar","Anne of Green Gables","Memoirs of a Geisha","Matilda","Charlie and the Chocolate Factory","Frankenstein","The Picture of Dorian Gray","Jane Eyre","Wuthering Heights","Les Misérables","War and Peace","Crime and Punishment","One Hundred Years of Solitude","Don Quixote","Dracula","The Odyssey","The Divine Comedy","The Adventures of Sherlock Holmes","The Martian","The Name of the Wind","The Silence of the Lambs","The Shining","Alice's Adventures in Wonderland"];
    const musicGenres = ["Rock","Pop","Hip-hop","Jazz","Blues","Country","Reggae","Electronic","R&B","Metal","Classical","Folk","Punk","Funk","Indie","Gospel","Techno","Soul","Ska","Rap"];
    const games = ["The Legend of Zelda: Breath of the Wild","The Witcher 3: Wild Hunt","Red Dead Redemption 2","Grand Theft Auto V","Minecraft","Fortnite","Super Mario Odyssey","Dark Souls III","Overwatch","Call of Duty: Modern Warfare","The Elder Scrolls V: Skyrim","God of War (2018)","Bloodborne","Animal Crossing: New Horizons","Halo: The Master Chief Collection","Destiny 2","Final Fantasy VII Remake","League of Legends","Super Smash Bros. Ultimate","Pokémon Sword and Shield","Doom Eternal","Cyberpunk 2077","Among Us","Apex Legends","Resident Evil 2 (2019)","Persona 5","Monster Hunter: World","Assassin's Creed Odyssey","Death Stranding","The Last of Us Part II","Sekiro: Shadows Die Twice","Fortnite Battle Royale","Genshin Impact","Rocket League","FIFA 21","Tom Clancy's Rainbow Six Siege","Valorant","Counter-Strike: Global Offensive","NBA 2K21","World of Warcraft","Among Us","Minecraft Dungeons","Fall Guys: Ultimate Knockout","Animal Crossing: New Leaf","The Legend of Zelda: Ocarina of Time","Super Mario 64","Call of Duty: Warzone","Ghost of Tsushima","Days Gone","Borderlands 3"];
    const education = ["High school diploma","Doctorate","Associate's degree","Bachelor's degree","Master's degree"];
    const lang = ["Mandarin Chinese","Spanish","English","Hindi","Arabic","Bengali","Portuguese","Russian","Japanese","Lahnda","German","Javanese","Korean","French","Telugu","Italian"];
    const dishes = ["Carbonara pasta","Sushi","Pizza","Steak","Hummus with vegetables","Pad Thai","Rice with vegetables and chicken (or teriyaki)","Hamburger","Taco","Spaghetti Bolognese","Chicken curry","Lasagna","Fried rice","Caesar salad","Fish and chips","Beef stir-fry","Lobster bisque","Beef Wellington","Chicken Alfredo","Tom yum soup"];
    const drinks = ["Water","Tea","Coffee","Juice","Milk","Lemonade","Soda","Iced tea","Smoothie"];
    const desserts = ["Chocolate cake","Cheesecake","Ice cream","Tiramisu","Fruit salad","Crème brûlée","Chocolate mousse","Panna cotta","Vanilla pudding","Chocolate brownies","Berry pie","Lemon pie","Chocolate lava cake","Cottage cheese pastry","Strawberry cheesecake","Caramel pudding","Apple pie","Caramel orange cake","Creamy fruit pancakes","Chocolate truffles"];
    const styles = ["Casual","Formal","Business Casual","Streetwear","Bohemian","Vintage","Hipster","Preppy","Gothic","Punk","Retro","Chic","Minimalist","Sporty","Tomboy","Feminine","Masculine","Urban","Elegant","Classic","Rocker","Artsy","Edgy","Avant-Garde","Glamorous","Ethnic","Eclectic","Surfer","Rugged","Western","Military","Steampunk","Cyberpunk","Geek","Nerd","Athleisure","Romantic","Gothabilly","Pin-up","Kawaii","Harajuku","Cosplay","Rockabilly","Mod","Hippie","Grunge","Emo","Rave","Techno","Boho-Chic","Art Deco","Art Nouveau","Victorian","Regency","Edwardian","Baroque","Rococo","Renaissance","Medieval","Tudor","Cottagecore","Dark Academia","Light Academia","Royalcore","Street Style","High Fashion","Beachwear","Casual Chic","Smart Casual","Eveningwear","Clubwear","Lingerie","Uniform","Traditional","Cultural","Festival","Costume","Vintage Inspired","Modern","Contemporary"];
    const powers = ["Reality warping","Energy absorption","Technopathy (ability to control technology with the mind)","Probability manipulation","Astral projection","Memory manipulation","Gravity control","Size manipulation","Elemental manipulation (control over elements like fire, water, earth, and air)","Animal communication","Plant manipulation","Telepathic mimicry (ability to copy and use others' powers temporarily through telepathy)","Sonic scream","Time travel","Molecular manipulation","Intangibility (ability to phase through objects)","Precognition (seeing the future)","Retrocognition (seeing the past)","Pyrokinesis (ability to control fire with the mind)","Hydrokinesis (ability to control water with the mind)"];
    const places = ["Machu Picchu, Peru","Statue of Liberty, New York, USA","Pyramids of Giza, Egypt","Strasbourg, France","Venice, Italy","Sydney Opera House, Australia","Saint Petersburg, Russia","Grand Canyon, Arizona, USA","Eiffel Tower, France","Great Wall of China"];
    const movieGenres = ["Action","Comedy","Drama","Horror","Science Fiction","Romance","Thriller","Adventure","Fantasy","Animation"];
    const personalityTypes = ["Leader, extrovert","Analyst, introvert","Adventurer, extrovert","Leader, introvert","Adventurer, introvert","Analyst, extrovert"];
    const relStatuses = ["Single","In a relationship","Married"];
    const actLevels = ["Active","Moderate","Sedentary"];

    const fields = [
        { id: "name", label: "Name", type: "text" },
        { id: "age", label: "Age", type: "number" },
        { id: "height", label: "Height", type: "number" },
        { id: "weight", label: "Weight", type: "number" },
        { id: "date_of_registration", label: "Date of registration", type: "date" },
        { id: "hobbies", label: "Hobbies", type: "text" },
        { id: "favorite_color", label: "Favorite color", type: "text" },
        { id: "favorite_pet", label: "Favorite pet", type: "text" },
        { id: "relationship_status", label: "Relationship status", type: "text" },
        { id: "favorite_book", label: "Favorite book", type: "text" },
        { id: "music", label: "Music", type: "text" },
        { id: "favorite_video_game", label: "Favorite video game", type: "text" },
        { id: "education_level", label: "Education level", type: "text" },
        { id: "language", label: "Language", type: "text" },
        { id: "favorite_dish", label: "Favorite dish", type: "text" },
        { id: "favorite_drink", label: "Favorite drink", type: "text" },
        { id: "favorite_dessert", label: "Favorite dessert", type: "text" },
        { id: "activities", label: "Activities", type: "text" },
        { id: "activity_level", label: "Activity level", type: "text" },
        { id: "fashion_style", label: "Fashion style", type: "text" },
        { id: "places_to_visit", label: "Places to visit", type: "text" },
        { id: "movie_preferences", label: "Movie preferences", type: "text" },
        { id: "personality_type", label: "Personality type", type: "text" },
        { id: "superhero_power", label: "Superhero power", type: "text" },
    ];
</script>

Enterprise applications often need to analyze datasets containing hundreds of thousands or even millions of records. Without efficient data processing, pivot tables can become slow and unresponsive, frustrating users who need quick insights from large volumes of data.

DHTMLX Pivot is designed to handle large datasets efficiently. This example demonstrates generating and loading datasets up to 1 million rows, with timing output to evaluate performance in your environment.

This example generates random datasets of configurable sizes (50K, 100K, 500K, 1M rows) and loads them into the pivot. A dropdown selector lets you switch between sizes, and timing statistics show both data generation time and pivot rendering time. The configuration groups data by hobbies (rows) and activities/language (columns) with min/max age, max height, and max weight aggregations.

Solution overview​

To load large datasets, create the pivot with an empty data array, then use setConfig({ data }) to inject generated data. Use requestAnimationFrame with setTimeout to measure the approximate rendering time after data injection.

Key points​

  • Scalable test range: This example benchmarks datasets from 50,000 to 1,000,000 rows so you can evaluate aggregation and rendering behavior in your environment
  • Dynamic data loading: Use setConfig({ data }) to inject or replace datasets at runtime - the pivot recalculates all aggregations and re-renders automatically
  • Performance measurement: Combine requestAnimationFrame with setTimeout to capture approximate rendering time after data injection for benchmarking
  • Virtual rendering: The pivot only renders visible rows in the DOM, keeping memory usage constant regardless of the total number of rows in the dataset

API reference​

Additional resources​