Want to know what happens backstage at Redpill Linpro?
Soon we can welcome you in!
Coming soon
Backstage
Sign up for our newsletter

Caching with user-specific data

Fri, 01/08/2016 - 10:44 -- Kristian Lyngstøl
One of the last news items to hit IT security in 2015 was Steam leaking credit card information between different users. For an outside observer, it looked like a classic example of leaking session data through bad cache policies.

Similar examples have been seen several times before. Here in Norway, "Kenneth (36)" was made famous when his tax report was leaked to thousands of other users just a few years ago, for example.

This is a guide on how to safe guard against such privacy disasters when dealing with cache and logged in users. Several examples for Varnish are provided, but the general strategy is somewhat universal to any cache.

Step 0: Mindset

It's important to established exactly what we are trying to protect against. If this was all about providing a configuration that worked right now, the problem would be easy. But what you are really trying to protect against is future mistakes made by yourself or other people two years down the road.

You need to make sure that you are thinking "This doesn't alter the content delivered from the backend, so I'll strip it from the request", instead of "This doesn't alter the content delivered from the backend, so I'll enforce caching". The latter is a dangerous assumption that will, eventually, come back to bite you.

When I hold Varnish training courses, I always ask the participants to think about what they prefer: Do they prefer disabling caching by accident, possibly taking down the site due to load issues, or do they prefer a fully operational site, delivering sensitive data to the wrong people?

You can also get both, but you can't chose "neither". Eventually someone makes a mistake, and how you planned for that will determine whether your boss is upset with you for a few hours, or whether your company ends up being international news and losing all trust of your users.

Step 1: Cookie policy

Varnish does not cache content requested by users with cookies by default. Unfortunately, this tends to break caching on almost every single site out there because practically everyone uses cookies. You can approach this challenge in a few different ways:

  1. Strip all cookies for content that you know should be cached. Don't cache content with cookies.
  2. Strip all cookies except for those you know you need. Add the cookie header to your cache hash. Override the default logic of Varnish to allow caching the content.
  3. Override the default logic of Varnish and force caching despite a Cookie header.

Avoid the last option at all cost. It WILL come back and bite you.

For sites where most content can be cached, option number one is usually more than enough, and very safe. Even with logged in users, this should be done. A lot of content will still be shared between users (CSS, images, public posts, etc) and shouldn't require a cookie to render properly.

The second option is a fairly safe universal approach. This allows you to cache content that is user-specific, but you run the risk of having a large cache. This is best done on a per-path basis.

The last option is provided as an obvious "don't do this" approach. If your backend server requires the cookies, then you have to assume that they mean something - different cookie, different content. If they don't require it, then strip it instead.

It's also worth mentioning that using a session cookie for everything is maybe not the wisest decision. If part of your site looks different based on a theme, but is otherwise shared, then it might make sense to have a separate "theme" cookie. With a session cookie, every themed resource would have to be cached individually for each user, but with a theme cookie, you only have to cache the themed resources for each different theme you provide. Similar logic can be applied to polls ("show poll/show result" based on whether the user has voted or not). Some of this logic can also be handled client side too, of course.

Step 2: Clean cookies

There are two ways to clean cookies in Varnish: Using regular expressions (which will cause significant hair loss if you have modify it), or using the cookie vmod (https://github.com/lkarsten/libvmod-cookie).

The cookie VMOD allows you to easily clean up the Cookie header.

Here's an example of how to filter away all irrelevant cookies:

import cookie;
sub vcl_recv {
      cookie.parse(req.http.cookie);
      cookie.filter_except("sessioncookie,themechoice");
      set req.http.cookie = cookie.get_string();
}

The above snippet will strip away all cookies except the ones named sessioncookie and themechoice. If either of those are present, caching is still not possible, though.

Step 3: Be selective

Start out by simply distinguishing what is relevant for a logged-in user and what isn't.

If you are starting out from scratch, now might be a good time to think about your URL name space. If you can use the URL to distinguish where cookies are relevant and not, it will save you a lot of trouble. For example:

import cookie;

sub vcl_recv {
        cookie.parse(req.http.cookie);
        cookie.filter_except("sessioncookie,themechoice");
        set req.http.cookie = cookie.get_string();
        if (req.url !~ "^/usercontent") {
                unset req.http.cookie;
        }
}

Now you have a VCL that caches all content for non-logged in users, but keeps the cookie for logged in users. Logged in users will then by-pass the cache, but only under the url "/usercontent" (e.g. "http://example.com/usercontent/profile").

If you have certain options on your site that change the style, for example, you can add an other dimension:

import cookie;
sub vcl_recv {
        cookie.parse(req.http.cookie);
        cookie.filter_except("sessioncookie,themechoice");
        set req.http.cookie = cookie.get_string();
        if (req.url ~ "^/theme/") {
                cookie.filter_except("themechoice");
                set req.http.cookie = cookie.get_string();
        } elsif (req.url !~ "^/usercontent") {
                unset req.http.cookie;
        }
}

The above will keep the theme-cookie for content under "/theme/", and the theme-cookie and session-cookie for content under "/usercontent".

An important note is the fail safe. The basic filtering is done outside of any if ()-clause. This is intentional, even if it might be slightly less optimal. Over time, your VCL might change, and having the basic filter outside of any if-clause ensures that it's always run, even after the if-clause is erroneously altered 18 months later.

Step 4: Make the cookie a part of the hash

To enable caching with cookies present, you will want to do two more things:

  1. Add the cookie header to the cache hash.
  2. Override the default logic.

The first part will ensure that to retrieve the content, you need the same cookie(s) as the original request.

Normally there are two things that makes a cached object unique: The path and the host. E.g: http://example.com/foo is distinct from http://example.com/bar. Adding the cookie header to this logic is done in vcl_hash. The original vcl_hash built-in looks like this:

sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}

To add the cookie header, all you need to do is add the following snippet to your VCL:

sub vcl_hash {
hash_data(req.http.Cookie);
}

This will execute before the built-in version, it doesn't override it, just adds to it.

If you are going to do ONE change after reading this blog post, adding that line should be it. If you do not clean up your cookies, you will have a very large cache of mostly duplicates, but you will have safe guarded yourself against a large amount of possible mistakes.

To override the default policy of not caching when cookies are used, you have to modify vcl_recv. It's worth reading the built-in VCL that you are about to override first. It's usually shipped in /usr/doc/varnish/builtin.vcl. Here's the unaltered one for Varnish 4.1:

sub vcl_recv {
if (req.method == "PRI") {
/* We do not support SPDY or HTTP/2.0 */
return (synth(405));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}

if (req.method != "GET" && req.method != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
return (hash);
}

Here's an example of an altered version, including the previous snippets for cleaning cookies and the vcl_hash snippet:

import cookie;

sub vcl_recv {
cookie.parse(req.http.cookie);
cookie.filter_except("sessioncookie,themechoice");
set req.http.cookie = cookie.get_string();
if (req.url ~ "^/theme/") {
cookie.filter_except("themechoice");
set req.http.cookie = cookie.get_string();
} elsif (req.url !~ "^/usercontent") {
unset req.http.cookie;
}

/*
* Modified builtin VCL follows:
*/
if (req.method == "PRI") {
/* We do not support SPDY or HTTP/2.0 */
return (synth(405));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}

if (req.method != "GET" && req.method != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
/*
* Allow caching with cookies, but not basic
* authentication.
*/
if (req.http.Authorization) {
/* Not cacheable by default */
return (pass);
}
return (hash);
}

sub vcl_hash {
hash_data(req.http.Cookie);
}

The change is small, but important. Look towards the very end of vcl_recv. This VCL will allow you to safely cache content even with cookies. Note how there are NO if-clauses in vcl_hash. This is, again, a safety mechanism. It doesn't cost you any thing when things work right, and when they don't work right, it ensures that only the matching user gets the content.

Step 5: Last but not least: Set-Cookie

We have covered how to deal with incoming cookies from browsers, but the biggest one remains: Set-Cookie.

This should be easy, but backend servers are frequently quite dumb, and the fix is typically applied the wrong way.

The Set-Cookie header is a response header sent by an application server of some sort. This is typically how a client gets a session cookie. One of the worst mistakes you can make is caching content with a set-cookie header present. This will result in the first user logging in just fine, then the next user tries to log in, but gets the cached response of the previous user, and you now have two people logged in as the same user.

This is also very easy to miss during testing, because you are just one person. Make sure you test with different browsers (at the same time).

Luckily, this is really simple to avoid:

Then Don't Do That Then.

Do not cache content with Set-Cookie headers present. Varnish doesn't do it by default (again, this is part of the built-in VCL) and you shouldn't either. If your application server sends Set-Cookie on every request without it, you should do three things:

  1. Strip Set-Cookie except for the actual login response page/API or similar.
  2. Fix your application.
  3. Not cache any response with Set-Cookie present.

An alternative could be:

sub vcl_backend_response {
if (bereq.url !~ "^/usercontent/login-api") {
unset beresp.http.set-cookie;
}
}

If you override the default VCL, remember to retain the logic regarding Set-Cookie. For example:

sub vcl_backend_response {
(other logic)
if (beresp.http.Set-Cookie) {
set beresp.uncacheable = true;
}
return (deliver);
}

This applies even if you explicitly strip the Set-Cookie header. The consequences of accidentally caching a Set-Cookie header is so disastrous that having more than one safety net is very important.

So the following is perfectly sensible:

sub vcl_backend_response {
if (bereq.url !~ "^/usercontent/login-api") {
unset beresp.http.set-cookie;
} else {
set beresp.uncacheable = true;
}
if (beresp.http.Set-Cookie) {
set beresp.uncacheable = true;
}
}

Even if it might look suboptimal, that VCL is likely going to be changed by multiple different people over a period of years. If you leave a generic fail-safe (with a strong comment of never to remove it) at the end, then accidentally fouling up the more specific logic further up isn't going to cause a privacy disaster.

Consider the following example:

sub vcl_backend_response {
if (bereql.url ~ "^/static") {
unset beresp.http.Set-Cookie;
} else {
set beresp.uncacheable = true;
}
if (beresp.http.Set-Cookie) {
set beresp.uncacheable = true;
}
return (deliver);
}

There is no way to reach the second if-clause with a Set-Cookie header intact and beresp.uncacheable not already being set to true. So what's the point?

Now imagines there's a second thing you need to cache some time later:

sub vcl_backend_response {
if (bereql.url ~ "^/static") {
unset beresp.http.Set-Cookie;
} else {
set beresp.uncacheable = true;
}
if (bereq.url ~ "^/js") {
set beresp.uncacheable = false;
set beresp.ttl = 10m;
}
if (beresp.http.Set-Cookie) {
set beresp.uncacheable = true;
}
return(deliver);
}

Without the extra check at the end, you would have opened up paths under "/js" for caching with Set-Cookie headers. Instead, you now have two conflicting if-clauses, where the latter one safe guards you from leaking Set-Cookie headers by preventing cache.

Summary

It's possible to cache content with logged in users and for logged in users, and do it safely despite operational changes that happen over time.

The most important approach to caching with logged in users is multiple levels of fail safes. Your site is never going to be done, and changes are going to be sporadic over time, most likely applied by different people. Fail safes that are redundant today, might be the only saving you from a nasty information leak in the future.

The basic strategy should be:

  1. Think long-term fail-safe VCL. You are safe guarding yourself more against future mistakes than present.
  2. Well-defined behavior, if possible. "This url-base is cached, this is static content, this is user-specific."
  3. Use the cookie VMOD to filter out cookies you don't explicitly care about.
  4. Add the Cookie-header to the cache hash.
  5. Never cache content with Set-Cookie. Strip away the response header and leave the default behavior if possible.
  6. Multiple fail safes. Assume that someone will change something they don't quite understand sixth months down the line.

And keep in mind: If Steam can mess this up, then so can you. Your site is never finished, so write configuration files that anticipate future changes.