Tag: filtering

Devil in the Details: Understanding Level of Detail Equations and Filtering

Tableau introduced level of detail (LODs) equations with Tableau 9.0, and with it came a huge number of new use cases.  LODs are an easy alternative to table calculations when looking for two-pass aggregations.  They don’t depend (much) on your viz layout, making them more easily reusable than table calcs.  One of the coolest pieces of functionality, however, is the flexibility provided with filters.  The addition of FIXED calculations gave Context filters additional functionality by adding something between Context filters and Dimension filters in the Tableau Order of Operations.

In short, the additional functionality could be summarized by two sentences.

  1. FIXED calculations compute before filters.
  2. Context filters still impact FIXED equations.

For most use cases, this holds true:  FIXED equations gave us a new layer of computation between filter levels.  If you play around with filters for long enough, however, you’ll note that there are times this isn’t true.  Take the below example.  I’ve written three calcs.

  1. SUM([Sales])
  2. {FIXED [Region] : SUM([Sales])}
  3. {FIXED : SUM([Sales])}

With nothing else on the viz, these all return the same results, as expected.  Filtering on State, however, returns surprising results.

When we filter to just one State (Alabama), our calculations return 3 different results.  Have we been lied to?!

Even though Dimension filters come after FIXED expressions in the order of operations, we can see that a Dimension filter has impacted the results of a FIXED equation…but not both FIXED equations.  What’s happened here?  The short answer is that our filter filtered out the entirety of a Region, and we have an equation FIXED on Region.  The longer answer requires understanding how LODs work and a look at the SQL issued by FIXED calculations.

At its core, Tableau takes your drag-and-drop commands and turns them into SQL.  There are tons of great resources of Tableau for the SQL writer available, so we’ll look specifically at the SQL issued by FIXED commands here.

When we drag a normal metric to a visualization, Tableau issues a query which is SELECT SUM(Metric) GROUP BY <list of dimensions on viz>.  The combination of dimensions on the visualization is what we call the “visualization level of detail” and determines the granularity of displayed data.  Declaring dimensions in an LOD allows us to create a different level of detail for the calculation itself.  At the end of the day, of course, the LOD is displayed on a chart, so we need to reconcile the viz level of detail with the calculation level of detail.  In this case, our Region calc is at a more granular level than our viz, so Tableau aggregates up for us.  Looking at the SQL just for that calculation, we see the below.  (I’ve cleaned up the SQL just because Tableau has some funny re-aliasing that goes on.  This isn’t verbatim what Tableau issues, but it’s the same query.)

SELECT SUM(t1.LODCalc)
FROM (
   SELECT Region AS Region,
   SUM(Sales) AS LODCalc FROM Orders 
   GROUP BY Region
) t1

Tableau first selects our Sales per Region, then aggregates them all together.  Because SUM is an additive metric, it returns the same as simply doing SUM(Sales).  Now let’s look at the query when we add SUM(Sales) to the viz.

SELECT t2.LODCalc , t0.Sales
FROM (
   SELECT SUM(Sales) AS Sales
   FROM Orders
) t0
CROSS JOIN (
   SELECT SUM(t1.Metric) AS LODCalc
   FROM (
      SELECT Region, SUM(Sales) AS Metric
      FROM Orders
      GROUP BY Region
   ) t1
) t2

Again, let’s walk through it step by step.  First, Tableau creates a table of sales by region (and names it t1).  Because our viz is at the full-table level, we don’t need this data disaggregated at all, so Tableau aggregates that up to a single row (and names it t2).  Then, Tableau cross-joins that to the SUM(Sales) total.  It results in a one-row table.

Now let’s look at it as a real use case.  I want to know the average sales in the Region for each State.  It’s simply {FIXED [Region} : AVG([Sales])}.

SELECT State, SUM(t1.RegionalAvg) 
FROM (
   SELECT Region, State
   FROM Orders
   GROUP BY Region, State
) t0
 INNER JOIN ( 
    SELECT Region, AVG(Sales) AS RegionalAvg
    FROM Orders 
    GROUP BY Region
) t1 ON t1.Region = t0.Region
GROUP BY State

This is where the important step first surfaces.  When we use the LOD on a viz, we inner join the LOD portion back to the original data set on the dimension which we FIXED to.  It’s the same step-by-step process as the SQL statements before.

  1. Create a table of AVG(Sales) per Region
  2. Join it back to our other data (in this case, just State name) on the appropriate dimension (in this case, Region)

In this case, we’re joining back on Region = Region.  If we filter down to a specific state, however, the join won’t return the data from any other Regions.  Our INNER JOIN on Region = Region now only returns data from the specific State we filtered to.  In this case, though, we only needed data from the South, because that’s all that’s being displayed on our viz.

If we weren’t looking at it at the State level, however, Tableau still loses the data.  Let’s go back to the original chart with 3 bars.  This time, I’ll filter just to Alabama.

The FIXED SUM(Sales) bar still returns our total sales.  SUM(Sales) returns just the data from Alabama, as we expect.  The FIXED Region calc, however, returns sales from the entire region that Alabama is a part of.  Let’s look at the query for SUM(Sales) and FIXED Region together.

SELECT t3.RolledUp, t0.Sales 
FROM(
  SELECT SUM(Sales) AS Sales
  FROM Orders
  WHERE State = 'Alabama'
) t0
   CROSS JOIN ( 
    SELECT SUM(t2.RegionalTotal) AS RolledUp
    FROM(
     SELECT Region
     FROM Orders
     WHERE State = 'Alabama'
     GROUP BY Region
   ) t1
  INNER JOIN (
  SELECT Region, SUM(Sales) AS RegionalTotal
  FROM Orders
  GROUP BY Region
) t2 ON t1.Region = t2.Region
) t3

Again, we can walk through this step by step.  First, Tableau builds a table of Sales by Region (t2).  Next, it joins that on to a list of our available states (t1) on Region = Region.  In this case, we only have one state (Alabama) and therefore only one region (South).  This is where the dimension filter impacts our LOD.  We’ve computed the sales for each region…but when we start to bring it back to the viz, the data gets excluded by a simple dimension filter.

Circling back to the original point…is it fair to say that a FIXED equation isn’t impacted by a normal dimension filter?  For most use cases, yes it is.  If, however, we filter out (whether explicitly or coincidentally) an entire member of the dimension we’ve fixed to, Tableau won’t have any way to bring that data back.  The whole concept is built on INNER JOINS, so we need the dimension to be present in our resulting data.  Relying on FIXED equations to cheat your dimension filters is great as long as you’ve vetted your use case, but it isn’t a bullet-proof approach to LOD filtering.

Date Filter Extensions in Tableau

Tableau handles dates fairly flexibly – it allows a variety of input formats, handles them as hierarchies naturally, and provides a ton of calculation flexibility for any sort of math or logic.  The two major criticisms I hear from Tableau users are how they interact with a) parameters and b) filters.  The parameter ask is fairly simple – people want parameters to automatically update to “today” when they load a dashboard.  Luckily, there’s a free extension available in the extensions gallery which does exactly this.  It’s built and hosted by Tableau, and the source code is freely available.

Filters, on the other hand, have a lot more options for implementation.  Relative date filters are very powerful, but what if I want to look at a subset of data from last year?  Date sliders are the only way to do that, but they have their own limitations.  A date slider only has two settings for it’s default settings.

  1. Full extent of the data
    1. When the dashboard loads, Tableau finds the full range of your date data and sets the bounds to those dates.  This is great because when you load the dashboard, it’ll take into account all of the most recent data.  The problem arises when you have 5 years worth of data.  Do you really want to query all that when it first loads, just so you can look at this week’s data?
  2. Pre-set values
    1. If you want to make sure that Tableau doesn’t query all of your data, you can pre-set the slider values.  The problem here is that it isn’t forward-looking.  If I publish a dashboard on 1/1/2019 and hardcode the values, then when someone loads the dashboard in the future, the filter will still be set to 1/1/2019 as the max.

So there’s the problem – Tableau allows you to hardcode both sides of of the slider or neither.  Most people want to hardcode one side and have the other be dynamic…so that’s the extension we built.  A Tableau Extension is a custom webpage which is added to a dashboard to extend the functionality.  For more info on extensions, check out the Tableau developers page.

Below is a quick tutorial to build out an extension for custom date functionality in Tableau.  The code I’m referencing is available here.  The left-hand side of that page contains an HTML file, a JS file, a TREX file, and a favicon file, which is everything you’ll need to create an extension.

  1. Build a webpage.
    1. An extension is a custom HTML page.  In this case, the HTML page does almost nothing, so it’ll just contain a title and references to the JS we’re using (jQuery, bootstrap, the Extensions Library, our JS file).
    2.  
  2. Add JS functionality.  Everything from here on out is in the JS file.
    1. First thing we need to do is initialize the API (line 1).
  3. Create a function to update your filter (line 5, updateFilterRange()).
  4. Set variables for your start and end points.
    1. We’ll hardcode the starting date to a date of your choice.  This is easily done by creating a date in JS.
      1. let minDate = new Date(“1/1/2014”)
    2. For the upper limit, you’ve got flexibility.  The two major use cases I see here are setting it to “today” or setting it to the highest date in the dataset.  Setting it to “today” is easy, as a Date instance in JS will default to “today”.  The other option is getting data from the workbook itself, which is what we’ve done in the sample code.
      1. Create a calculation in the workbook which returns your highest date.  Using an LOD to return this is simple {FIXED : MAX([Date])}.  Put this calculation on detail on your target worksheet to make it available to the API.
      2. In your JS, we make an API call to return data from your sheet.  This returns summary data, meaning one result per mark on the sheet.  From the table returned, we’ll find your Max Date column.  Because it returns the same result for every mark, we’ll can use Max Date from any mark.  (Note that we could have calculated max date here instead of using an LOD, but we’re taking advantage of the Hyper database instead of iterating through all of your data in JS).  Set this as your maxDate variable.
      3. Invoke the applyFilterRangeAsync function.  It takes 3 arguments: the field you defined at the beginning, the minDate that you hard-coded, and the maxDate returned from our table.
  5. Call the function you’ve created.

tableau.extensions.initializeAsync().then(() => {
  updateFilterRangeDataSource();
});

function updateFilterRangeDataSource() {
  //define our target sheet, field, minimum date, and dashboard
  let fieldName = 'Date';
  let sheetName = 'Timeline';
  let minDate = new Date("1/1/2014");
  let dashboard = tableau.extensions.dashboardContent.dashboard;
  let selectedWorksheet = dashboard.worksheets.find(w => w.name === sheetName);
  //get data back from the workbook to find our highest possible date
  selectedWorksheet.getSummaryDataAsync().then( table => {
    let maxDateColumn = table.columns.find(columnNames => columnNames.fieldName === 'Max Date').index;
    let maxDate = new Date(table.data[0][maxDateColumn].value);
    //make the API call to update the Tableau Filter
    selectedWorksheet.applyRangeFilterAsync(fieldName, { min: minDate ,max: maxDate});
  });
}

6. Add the TREX file to your workbook!

There it is!  15 lines of JavaScript and you’ve built a feature which massively extends Tableau’s default date functionality.  Of course, this is far from the limits of what we can do with the extensions.  If you’d like to build a custom UI for your date filters, there are tons of JS widgets you could incorporate into an extension.  If you’d like any other default ranges, you just need to tweak the variables.  If you’d like to make this more scalable, you can use the Settings namespace in the Extensions API to make this workbook-agnostic.

Overall, though, it’s cool to note what we’ve done here.  The Extensions API is often billed as a way to integrate two applications, or a way to create custom visual elements (and it’s great for both of these things).  To me, however, the biggest wins I’ve had with the Extensions API have been creating functionality that Tableau previously didn’t have, and doing so in an efficient way.