Azure Information Explorer: Log and telemetry analytics benchmark | Azure Weblog and Updates


    Azure Information Explorer (ADX), a part of Azure Synapse Analytics, is a extremely scalable analytics service optimized for structured, semi-structured, and unstructured knowledge. It gives customers with an interactive question expertise that unlocks insights from the ocean of ever-growing log and telemetry knowledge. It’s the good service to investigate excessive volumes of recent and historic knowledge within the cloud by utilizing SQL or the Kusto Question Language (KQL), a strong and user-friendly question language.

    Azure Information Explorer is a key enabler for Microsoft’s personal digital transformation. Just about all Microsoft services use ADX in a method or one other; this consists of troubleshooting, prognosis, monitoring, machine studying, and as a knowledge platform for Azure companies resembling Azure Monitor, PlayFab, Sentinel, Microsoft 365 Defender, and plenty of others. Microsoft’s clients and companions are utilizing ADX for a big number of eventualities from fleet administration, manufacturing, safety analytics options, bundle monitoring and logistics, IoT gadget monitoring, monetary transaction monitoring, and plenty of different eventualities. During the last years, the service has seen phenomenal development and is now working on hundreds of thousands of Azure digital machine cores.

    Final 12 months, the third era of the Kusto engine (EngineV3) was launched and is at present supplied as a clear, in-place improve to all customers not already utilizing the newest model. The brand new engine incorporates a fully new implementation of the storage, cache, and question execution layers. Consequently, efficiency has doubled or extra in lots of mission-critical workloads.

    Superior efficiency and cost-efficiency with Azure Information Explorer

    To raised assist our customers assess the efficiency of the brand new engine and value benefits of ADX, we regarded for an present telemetry and logs benchmark that has the workload traits widespread to what we see with our customers:

    1. Telemetry tables that include structured, semi-structured, and unstructured knowledge sorts.
    2. Data within the a whole lot of billions to check large scale.
    3. Queries that signify widespread diagnostic and monitoring eventualities.

    As we didn’t discover an present benchmark to satisfy these wants, we collaborated with and sponsored GigaOm to create and run one. The brand new logs and telemetry benchmark is publicly out there on this GitHub repo. This repository features a knowledge generator to generate datasets of 1GB, 1TB, and 100TB, in addition to a set of 19 queries and a check driver to execute the benchmark.

    The outcomes, now out there within the GigaOm report, present that Azure Information Explorer gives superior efficiency at a considerably decrease value in each single and high-concurrency eventualities. For instance, the next chart taken from the report shows the outcomes of executing the benchmark whereas simulating 50 concurrent customers: 

    A column chart comparing Google BigQuery, Snowflake, and Azure Data Explorer query execution times. The measurement is the sum of average query runs for all 19 queries by 50 concurrent users. The chart shows the following measurements: Google BigQuery 1159.15 Seconds, Snowflake 1238.47 seconds, and Azure Data Explorer 54.52 seconds.

    Be taught extra

    For additional insights, we extremely suggest studying the full report. And don’t simply take our phrase for it. Use the Azure Information Explorer free providing to load your knowledge and analyze it at excessive velocity and unmatched productiveness.

    Try our documentation to discover out extra about Azure Information Explorer and be taught extra about Azure Synapse Analytics. For deeper technical info, try the brand new e-book Scalable Information Analytics with Azure Information Explorer by Jason Myerscough.


    Please enter your comment!
    Please enter your name here