VCL
VCL is kind of horrible. Okay! We have that out of the way!
VCL is kind of horrible, but it's kind of horrible for understandable reasons. As engineers, it's very easy for us to roll up to a project all like...
...but, and I know it can be difficult: imagine the author(s) had some kind of constraints we're perhaps unaware of. In this instance, the first version of Varnish was released in 2006, 18 years ago. Here are neat facts about the year 2006:
- Linux Containers didn't exist, let alone Docker, Kubernetes, etc.
- AWS EC2 was a brand new product. CloudFront, Amazon's CDN product, wasn't launched until 2008.
- WebAssembly? Not for another 11 years.
- This author lived in a building that was previously a pants factory in Harrisonburg, Virginia.
Now let's say that it's circa 2005-2006, and we're trying to build a reverse proxy daemon that allows a substantial degree of custom logic in how it handles requests and responses. Popular programming languages at the time included C, Java, PHP, and Visual Basic.
It needs to be fast, so interpreted languages like PHP are out of the question. It needs to be cross-platform, so Visual Basic is a non-starter, even in its compiled form. Java's maybe fast-enough at the time, but now we need to write the whole thing in Java, which comes with the attendant complexity of having users install the JVM, etc. C is plenty-fast, and we're already writing the rest of the daemon in C, but it's a language that's terribly easy to make serious mistakes in.
What's a person to do? Well, defining a small domain-specific language that is transpiled to C and literally fed into the C compiler to create a shared object that is loaded into the daemon is not actually the most unhinged idea ever, given the time period and the constraints. This is what Varnish does under the hood–it takes VCL, turns it into C code, and then compiled that into a shared library that is loaded into the daemon.
No, really.
Let's take a Docker container definition for a version of Varnish close to what Fastly forked from and see if we can get it to spit out some C.
Don't try this at home. If you're going to, though, the package link in the
above Dockerfile doesn't work anymore, and you'll need to change it to a
mirror that still works.
It also isn't happy compiling under the aarch64 architecture that may be the
one your machine is using, so you'll need to force amd64 by changing the
FROM instruction in the Dockerfile to --platform=linux/amd64 ubuntu:14.04.
Time rots all bits and such.
Here's a little bit of VCL cobbled together from
the Varnish examples.
It strips the Google Analytics query string parameters from URLs in vcl_recv,
because otherwise we may both duplicate cached content and cause unnecessary
cache MISSes.
backend default {
.host = "localhost";
.port = "80";
}
sub vcl_recv {
if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
set req.url = regsub(req.url, "\?&", "?");
set req.url = regsub(req.url, "\?$", "");
}
}
Now lets run varnishd with the -C flag to get it to emit the transpiled C:
$ docker run --platform linux/amd64 -v "$HOME/tmp/example.vcl":/tmp/example.vcl varnish-fastly varnishd -C -f /tmp/example.vcl
Voila. We have the C code Varnish would compile under the hood:
/*
* $Id$
*
* NB: This file is machine generated, DO NOT EDIT!
*
* Edit and run vcc_gen_fixed_token.tcl instead
*/
struct sess;
struct cli;
typedef void vcl_init_f(struct cli *);
typedef void vcl_fini_f(struct cli *);
typedef int vcl_func_f(struct sess *sp);
/* VCL Methods */
#define VCL_MET_RECV (1U << 0)
#define VCL_MET_PIPE (1U << 1)
#define VCL_MET_PASS (1U << 2)
#define VCL_MET_HASH (1U << 3)
#define VCL_MET_MISS (1U << 4)
#define VCL_MET_HIT (1U << 5)
#define VCL_MET_FETCH (1U << 6)
#define VCL_MET_DELIVER (1U << 7)
#define VCL_MET_ERROR (1U << 8)
#define VCL_MET_MAX 9
/* VCL Returns */
#define VCL_RET_DELIVER 0
#define VCL_RET_ERROR 1
#define VCL_RET_FETCH 2
#define VCL_RET_HASH 3
#define VCL_RET_LOOKUP 4
#define VCL_RET_PASS 5
#define VCL_RET_PIPE 6
#define VCL_RET_RESTART 7
#define VCL_RET_MAX 8
struct VCL_conf {
unsigned magic;
#define VCL_CONF_MAGIC 0x7406c509 /* from /dev/random */
struct director **director;
unsigned ndirector;
struct vrt_ref *ref;
unsigned nref;
unsigned busy;
unsigned discard;
unsigned nsrc;
const char **srcname;
const char **srcbody;
vcl_init_f *init_func;
vcl_fini_f *fini_func;
vcl_func_f *recv_func;
vcl_func_f *pipe_func;
vcl_func_f *pass_func;
vcl_func_f *hash_func;
vcl_func_f *miss_func;
vcl_func_f *hit_func;
vcl_func_f *fetch_func;
vcl_func_f *deliver_func;
vcl_func_f *error_func;
};
/*-
* Copyright (c) 2006 Verdens Gang AS
* Copyright (c) 2006-2009 Linpro AS
* All rights reserved.
*
* Author: Poul-Henning Kamp <phk@phk.freebsd.dk>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* $Id$
*
* Runtime support for compiled VCL programs.
*
* XXX: When this file is changed, lib/libvcl/vcc_gen_fixed_token.tcl
* XXX: *MUST* be rerun.
*/
struct sess;
struct vsb;
struct cli;
struct director;
struct VCL_conf;
struct sockaddr;
/*
* A backend probe specification
*/
extern const void * const vrt_magic_string_end;
struct vrt_backend_probe {
const char *url;
const char *request;
double timeout;
double interval;
unsigned exp_status;
unsigned window;
unsigned threshold;
unsigned initial;
};
/*
* A backend is a host+port somewhere on the network
*/
struct vrt_backend {
const char *vcl_name;
const char *ident;
const char *hosthdr;
const unsigned char *ipv4_sockaddr;
const char *ipv4_addr;
const unsigned char *ipv6_sockaddr;
const char *ipv6_addr;
const char *port;
double connect_timeout;
double first_byte_timeout;
double between_bytes_timeout;
unsigned max_connections;
unsigned saintmode_threshold;
struct vrt_backend_probe probe;
};
/*
* A director with an unpredictable reply
*/
struct vrt_dir_random_entry {
int host;
double weight;
};
struct vrt_dir_random {
const char *name;
unsigned retries;
unsigned nmember;
const struct vrt_dir_random_entry *members;
};
/*
* A director with round robin selection
*/
struct vrt_dir_round_robin_entry {
int host;
};
struct vrt_dir_round_robin {
const char *name;
unsigned nmember;
const struct vrt_dir_round_robin_entry *members;
};
/*
* A director with dns-based selection
*/
struct vrt_dir_dns_entry {
int host;
};
struct vrt_dir_dns {
const char *name;
const char *suffix;
const double ttl;
unsigned nmember;
const struct vrt_dir_dns_entry *members;
};
/*
* other stuff.
* XXX: document when bored
*/
struct vrt_ref {
unsigned source;
unsigned offset;
unsigned line;
unsigned pos;
unsigned count;
const char *token;
};
/* ACL related */
#define VRT_ACL_MAXADDR 16 /* max(IPv4, IPv6) */
void VRT_acl_log(const struct sess *, const char *msg);
/* Regexp related */
void VRT_re_init(void **, const char *);
void VRT_re_fini(void *);
int VRT_re_match(const char *, void *re);
const char *VRT_regsub(const struct sess *sp, int all, const char *,
void *, const char *);
void VRT_panic(struct sess *sp, const char *, ...);
void VRT_ban(struct sess *sp, char *, ...);
void VRT_ban_string(struct sess *sp, const char *, ...);
void VRT_purge(struct sess *sp, double ttl, double grace);
void VRT_log(struct sess *, const char *msg, ...);
void VRT_count(const struct sess *, unsigned);
int VRT_rewrite(const char *, const char *);
void VRT_error(struct sess *, unsigned, const char *);
int VRT_switch_config(const char *);
enum gethdr_e { HDR_REQ, HDR_RESP, HDR_OBJ, HDR_BEREQ, HDR_BERESP };
char *VRT_GetHdr(const struct sess *, enum gethdr_e where, const char *);
void VRT_SetHdr(const struct sess *, enum gethdr_e where, const char *,
const char *, ...);
void VRT_handling(struct sess *sp, unsigned hand);
/* Simple stuff */
int VRT_strcmp(const char *s1, const char *s2);
void VRT_memmove(void *dst, const void *src, unsigned len);
void VRT_ESI(struct sess *sp);
void VRT_Rollback(struct sess *sp);
/* Synthetic pages */
void VRT_synth_page(struct sess *sp, unsigned flags, const char *, ...);
/* Backend related */
void VRT_init_dir(struct cli *, struct director **, const char *name,
int idx, const void *priv);
void VRT_fini_dir(struct cli *, struct director *);
char *VRT_IP_string(const struct sess *sp, const struct sockaddr *sa);
char *VRT_int_string(const struct sess *sp, int);
char *VRT_double_string(const struct sess *sp, double);
char *VRT_time_string(const struct sess *sp, double);
const char *VRT_backend_string(struct sess *sp);
#define VRT_done(sp, hand) \
do { \
VRT_handling(sp, hand); \
return (1); \
} while (0)
/*
* $Id$
*
* NB: This file is machine generated, DO NOT EDIT!
*
* Edit and run vcc_gen_fixed_token.tcl instead
*/
struct sockaddr * VRT_r_client_ip(const struct sess *);
const char * VRT_r_client_identity(struct sess *);
void VRT_l_client_identity(struct sess *, const char *, ...);
struct sockaddr * VRT_r_server_ip(struct sess *);
const char * VRT_r_server_hostname(struct sess *);
const char * VRT_r_server_identity(struct sess *);
int VRT_r_server_port(struct sess *);
const char * VRT_r_req_request(const struct sess *);
void VRT_l_req_request(const struct sess *, const char *, ...);
const char * VRT_r_req_url(const struct sess *);
void VRT_l_req_url(const struct sess *, const char *, ...);
const char * VRT_r_req_proto(const struct sess *);
void VRT_l_req_proto(const struct sess *, const char *, ...);
void VRT_l_req_hash(struct sess *, const char *);
struct director * VRT_r_req_backend(struct sess *);
void VRT_l_req_backend(struct sess *, struct director *);
int VRT_r_req_restarts(const struct sess *);
double VRT_r_req_grace(struct sess *);
void VRT_l_req_grace(struct sess *, double);
const char * VRT_r_req_xid(struct sess *);
unsigned VRT_r_req_esi(struct sess *);
void VRT_l_req_esi(struct sess *, unsigned);
unsigned VRT_r_req_backend_healthy(const struct sess *);
unsigned VRT_r_req_hash_ignore_busy(struct sess *);
void VRT_l_req_hash_ignore_busy(struct sess *, unsigned);
unsigned VRT_r_req_hash_always_miss(struct sess *);
void VRT_l_req_hash_always_miss(struct sess *, unsigned);
const char * VRT_r_bereq_request(const struct sess *);
void VRT_l_bereq_request(const struct sess *, const char *, ...);
const char * VRT_r_bereq_url(const struct sess *);
void VRT_l_bereq_url(const struct sess *, const char *, ...);
const char * VRT_r_bereq_proto(const struct sess *);
void VRT_l_bereq_proto(const struct sess *, const char *, ...);
double VRT_r_bereq_connect_timeout(struct sess *);
void VRT_l_bereq_connect_timeout(struct sess *, double);
double VRT_r_bereq_first_byte_timeout(struct sess *);
void VRT_l_bereq_first_byte_timeout(struct sess *, double);
double VRT_r_bereq_between_bytes_timeout(struct sess *);
void VRT_l_bereq_between_bytes_timeout(struct sess *, double);
const char * VRT_r_beresp_proto(const struct sess *);
void VRT_l_beresp_proto(const struct sess *, const char *, ...);
void VRT_l_beresp_saintmode(const struct sess *, double);
int VRT_r_beresp_status(const struct sess *);
void VRT_l_beresp_status(const struct sess *, int);
const char * VRT_r_beresp_response(const struct sess *);
void VRT_l_beresp_response(const struct sess *, const char *, ...);
unsigned VRT_r_beresp_cacheable(const struct sess *);
void VRT_l_beresp_cacheable(const struct sess *, unsigned);
double VRT_r_beresp_ttl(const struct sess *);
void VRT_l_beresp_ttl(const struct sess *, double);
double VRT_r_beresp_grace(const struct sess *);
void VRT_l_beresp_grace(const struct sess *, double);
const char * VRT_r_obj_proto(const struct sess *);
void VRT_l_obj_proto(const struct sess *, const char *, ...);
int VRT_r_obj_status(const struct sess *);
void VRT_l_obj_status(const struct sess *, int);
const char * VRT_r_obj_response(const struct sess *);
void VRT_l_obj_response(const struct sess *, const char *, ...);
int VRT_r_obj_hits(const struct sess *);
unsigned VRT_r_obj_cacheable(const struct sess *);
void VRT_l_obj_cacheable(const struct sess *, unsigned);
double VRT_r_obj_ttl(const struct sess *);
void VRT_l_obj_ttl(const struct sess *, double);
double VRT_r_obj_grace(const struct sess *);
void VRT_l_obj_grace(const struct sess *, double);
double VRT_r_obj_lastuse(const struct sess *);
const char * VRT_r_resp_proto(const struct sess *);
void VRT_l_resp_proto(const struct sess *, const char *, ...);
int VRT_r_resp_status(const struct sess *);
void VRT_l_resp_status(const struct sess *, int);
const char * VRT_r_resp_response(const struct sess *);
void VRT_l_resp_response(const struct sess *, const char *, ...);
double VRT_r_now(const struct sess *);
/* ---===### VCC generated below here ###===---*/
extern const struct VCL_conf VCL_conf;
#define VGCDIR(n) VCL_conf.director[VGC_backend_##n]
#define VGC_backend__default 1
static const unsigned char sockaddr0[17] = {
16, /* Length */
2, 0, 0, 80, 146, 75, 77, 140,
0, 0, 0, 0, 0, 0, 0, 0
};
static const struct vrt_backend vgc_dir_priv__default = {
.vcl_name = "default",
.ipv4_sockaddr = sockaddr0,
.ipv4_addr = "146.75.77.140",
.port = "80",
.ident =
"backend default "
"{ "
". "
"host "
"= "
"\"" "localhost" "\" "
"; "
". "
"port "
"= "
"\"" "80" "\" "
"; "
"} ",
.hosthdr = "localhost",
.saintmode_threshold = -1,
};
static void *VGC_re_0;
static void *VGC_re_1;
static void *VGC_re_2;
static void *VGC_re_3;
static void *VGC_re_4;
#define VGC_NREFS 49
static int
VGC_function_vcl_recv (struct sess *sp)
{
/* ... from (input Line 6 Pos 5) */
{
{
VRT_count(sp, 1);
if
(
(
(
VRT_count(sp, 2),
(
VRT_re_match( VRT_r_req_url(sp), VGC_re_0)
)
)
)
)
{
VRT_count(sp, 3);
VRT_l_req_url(sp, VRT_regsub(sp, 1, VRT_r_req_url(sp), VGC_re_1, ""), vrt_magic_string_end);
VRT_l_req_url(sp, VRT_regsub(sp, 1, VRT_r_req_url(sp), VGC_re_2, "?"), vrt_magic_string_end);
VRT_l_req_url(sp, VRT_regsub(sp, 0, VRT_r_req_url(sp), VGC_re_3, "?"), vrt_magic_string_end);
VRT_l_req_url(sp, VRT_regsub(sp, 0, VRT_r_req_url(sp), VGC_re_4, ""), vrt_magic_string_end);
}
VRT_count(sp, 4);
}
}
/* ... from (Default Line 42 Pos 5) */
{
{
VRT_count(sp, 5);
if
(
(
(
VRT_count(sp, 6),
(
VRT_r_req_restarts(sp) == 0
)
)
)
)
{
VRT_count(sp, 7);
if
(
(
(
VRT_count(sp, 8),
(
VRT_GetHdr(sp, HDR_REQ, "\020x-forwarded-for:") != (void*)0
)
)
)
)
{
VRT_count(sp, 9);
VRT_SetHdr(sp, HDR_REQ, "\020X-Forwarded-For:", VRT_GetHdr(sp, HDR_REQ, "\020X-Forwarded-For:"), ", ", VRT_IP_string(sp, VRT_r_client_ip(sp)), vrt_magic_string_end);
}
else
{
VRT_count(sp, 10);
VRT_SetHdr(sp, HDR_REQ, "\020X-Forwarded-For:", VRT_IP_string(sp, VRT_r_client_ip(sp)), vrt_magic_string_end);
}
}
VRT_count(sp, 11);
if
(
(
(
VRT_count(sp, 12),
(
VRT_strcmp(VRT_r_req_request(sp), "GET")
)
) && (
VRT_count(sp, 13),
(
VRT_strcmp(VRT_r_req_request(sp), "HEAD")
)
) && (
VRT_count(sp, 14),
(
VRT_strcmp(VRT_r_req_request(sp), "PUT")
)
) && (
VRT_count(sp, 15),
(
VRT_strcmp(VRT_r_req_request(sp), "POST")
)
) && (
VRT_count(sp, 16),
(
VRT_strcmp(VRT_r_req_request(sp), "TRACE")
)
) && (
VRT_count(sp, 17),
(
VRT_strcmp(VRT_r_req_request(sp), "OPTIONS")
)
) && (
VRT_count(sp, 18),
(
VRT_strcmp(VRT_r_req_request(sp), "DELETE")
)
)
)
)
{
VRT_count(sp, 19);
VRT_done(sp, VCL_RET_PIPE);
}
VRT_count(sp, 20);
if
(
(
(
VRT_count(sp, 21),
(
VRT_strcmp(VRT_r_req_request(sp), "GET")
)
) && (
VRT_count(sp, 22),
(
VRT_strcmp(VRT_r_req_request(sp), "HEAD")
)
)
)
)
{
VRT_count(sp, 23);
VRT_done(sp, VCL_RET_PASS);
}
VRT_count(sp, 24);
if
(
(
(
VRT_count(sp, 25),
(
VRT_GetHdr(sp, HDR_REQ, "\016Authorization:") != (void*)0
)
)
) || (
(
VRT_count(sp, 26),
(
VRT_GetHdr(sp, HDR_REQ, "\007Cookie:") != (void*)0
)
)
)
)
{
VRT_count(sp, 27);
VRT_done(sp, VCL_RET_PASS);
}
VRT_count(sp, 28);
VRT_done(sp, VCL_RET_LOOKUP);
}
}
}
static int
VGC_function_vcl_pipe (struct sess *sp)
{
/* ... from (Default Line 72 Pos 5) */
{
{
VRT_count(sp, 29);
VRT_done(sp, VCL_RET_PIPE);
}
}
}
static int
VGC_function_vcl_pass (struct sess *sp)
{
/* ... from (Default Line 82 Pos 5) */
{
{
VRT_count(sp, 30);
VRT_done(sp, VCL_RET_PASS);
}
}
}
static int
VGC_function_vcl_hash (struct sess *sp)
{
/* ... from (Default Line 86 Pos 5) */
{
{
VRT_count(sp, 31);
VRT_l_req_hash(sp, VRT_r_req_url(sp));
if
(
(
(
VRT_count(sp, 32),
(
VRT_GetHdr(sp, HDR_REQ, "\005host:") != (void*)0
)
)
)
)
{
VRT_count(sp, 33);
VRT_l_req_hash(sp, VRT_GetHdr(sp, HDR_REQ, "\005host:"));
}
else
{
VRT_count(sp, 34);
VRT_l_req_hash(sp, VRT_IP_string(sp, VRT_r_server_ip(sp)));
}
VRT_done(sp, VCL_RET_HASH);
}
}
}
static int
VGC_function_vcl_miss (struct sess *sp)
{
/* ... from (Default Line 103 Pos 5) */
{
{
VRT_count(sp, 39);
VRT_done(sp, VCL_RET_FETCH);
}
}
}
static int
VGC_function_vcl_hit (struct sess *sp)
{
/* ... from (Default Line 96 Pos 5) */
{
{
VRT_count(sp, 35);
if
(
(
(
VRT_count(sp, 36),
!(
VRT_r_obj_cacheable(sp)
)
)
)
)
{
VRT_count(sp, 37);
VRT_done(sp, VCL_RET_PASS);
}
VRT_count(sp, 38);
VRT_done(sp, VCL_RET_DELIVER);
}
}
}
static int
VGC_function_vcl_fetch (struct sess *sp)
{
/* ... from (Default Line 107 Pos 5) */
{
{
VRT_count(sp, 40);
if
(
(
(
VRT_count(sp, 41),
!(
VRT_r_beresp_cacheable(sp)
)
)
)
)
{
VRT_count(sp, 42);
VRT_done(sp, VCL_RET_PASS);
}
VRT_count(sp, 43);
if
(
(
(
VRT_count(sp, 44),
(
VRT_GetHdr(sp, HDR_BERESP, "\013Set-Cookie:") != (void*)0
)
)
)
)
{
VRT_count(sp, 45);
VRT_done(sp, VCL_RET_PASS);
}
VRT_count(sp, 46);
VRT_done(sp, VCL_RET_DELIVER);
}
}
}
static int
VGC_function_vcl_deliver (struct sess *sp)
{
/* ... from (Default Line 117 Pos 5) */
{
{
VRT_count(sp, 47);
VRT_done(sp, VCL_RET_DELIVER);
}
}
}
static int
VGC_function_vcl_error (struct sess *sp)
{
/* ... from (Default Line 121 Pos 5) */
{
{
VRT_count(sp, 48);
VRT_SetHdr(sp, HDR_OBJ, "\015Content-Type:", "text/html; charset=utf-8", vrt_magic_string_end);
VRT_synth_page(sp, 0, "\n<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\n \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n<html>\n <head>\n <title>", VRT_int_string(sp, VRT_r_obj_status(sp)), " ", VRT_r_obj_response(sp), "</title>\n </head>\n <body>\n <h1>Error ", VRT_int_string(sp, VRT_r_obj_status(sp)), " ", VRT_r_obj_response(sp), "</h1>\n <p>", VRT_r_obj_response(sp), "</p>\n <h3>Guru Meditation:</h3>\n <p>XID: ", VRT_r_req_xid(sp), "</p>\n <hr>\n <p>Varnish cache server</p>\n </body>\n</html>\n", vrt_magic_string_end);
VRT_done(sp, VCL_RET_DELIVER);
}
}
}
static struct vrt_ref VGC_ref[VGC_NREFS] = {
[ 1] = { 0, 88, 7, 5, 0, "if" },
[ 2] = { 0, 92, 7, 9, 0, "req.url" },
[ 3] = { 0, 195, 8, 9, 0, "set" },
[ 4] = { 0, 584, 13, 3, 0, "}" },
[ 5] = { 1, 1832, 43, 5, 0, "if" },
[ 6] = { 1, 1836, 43, 9, 0, "req.restarts" },
[ 7] = { 1, 1858, 44, 9, 0, "if" },
[ 8] = { 1, 1862, 44, 13, 0, "req.http.x-forwarded-for" },
[ 9] = { 1, 1895, 45, 13, 0, "set" },
[ 10] = { 1, 1984, 48, 13, 0, "set" },
[ 11] = { 1, 2039, 51, 5, 0, "if" },
[ 12] = { 1, 2043, 51, 9, 0, "req.request" },
[ 13] = { 1, 2073, 52, 7, 0, "req.request" },
[ 14] = { 1, 2104, 53, 7, 0, "req.request" },
[ 15] = { 1, 2134, 54, 7, 0, "req.request" },
[ 16] = { 1, 2165, 55, 7, 0, "req.request" },
[ 17] = { 1, 2197, 56, 7, 0, "req.request" },
[ 18] = { 1, 2231, 57, 7, 0, "req.request" },
[ 19] = { 1, 2319, 59, 9, 0, "return" },
[ 20] = { 1, 2344, 61, 5, 0, "if" },
[ 21] = { 1, 2348, 61, 9, 0, "req.request" },
[ 22] = { 1, 2372, 61, 33, 0, "req.request" },
[ 23] = { 1, 2461, 63, 9, 0, "return" },
[ 24] = { 1, 2486, 65, 5, 0, "if" },
[ 25] = { 1, 2490, 65, 9, 0, "req.http.Authorization" },
[ 26] = { 1, 2516, 65, 35, 0, "req.http.Cookie" },
[ 27] = { 1, 2582, 67, 9, 0, "return" },
[ 28] = { 1, 2607, 69, 5, 0, "return" },
[ 29] = { 1, 3002, 79, 5, 0, "return" },
[ 30] = { 1, 3039, 83, 5, 0, "return" },
[ 31] = { 1, 3076, 87, 5, 0, "set" },
[ 32] = { 1, 3109, 88, 9, 0, "req.http.host" },
[ 33] = { 1, 3134, 89, 9, 0, "set" },
[ 34] = { 1, 3186, 91, 9, 0, "set" },
[ 35] = { 1, 3259, 97, 5, 0, "if" },
[ 36] = { 1, 3263, 97, 9, 0, "!" },
[ 37] = { 1, 3289, 98, 9, 0, "return" },
[ 38] = { 1, 3314, 100, 5, 0, "return" },
[ 39] = { 1, 3354, 104, 5, 0, "return" },
[ 40] = { 1, 3393, 108, 5, 0, "if" },
[ 41] = { 1, 3397, 108, 9, 0, "!" },
[ 42] = { 1, 3426, 109, 9, 0, "return" },
[ 43] = { 1, 3451, 111, 5, 0, "if" },
[ 44] = { 1, 3455, 111, 9, 0, "beresp.http.Set-Cookie" },
[ 45] = { 1, 3489, 112, 9, 0, "return" },
[ 46] = { 1, 3514, 114, 5, 0, "return" },
[ 47] = { 1, 3557, 118, 5, 0, "return" },
[ 48] = { 1, 3598, 122, 5, 0, "set" },
};
static void
VGC_Init(struct cli *cli)
{
VRT_init_dir(cli, VCL_conf.director, "simple",
VGC_backend__default, &vgc_dir_priv__default);
VCL_conf.director[0] = VCL_conf.director[1];
VRT_re_init(&VGC_re_0, "(\\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=");
VRT_re_init(&VGC_re_1, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\\-\\.%]+)");
VRT_re_init(&VGC_re_2, "\\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\\-\\.%]+)");
VRT_re_init(&VGC_re_3, "\\?&");
VRT_re_init(&VGC_re_4, "\\?$");
}
static void
VGC_Fini(struct cli *cli)
{
VRT_fini_dir(cli, VGCDIR(_default));
VRT_re_fini(VGC_re_0);
VRT_re_fini(VGC_re_1);
VRT_re_fini(VGC_re_2);
VRT_re_fini(VGC_re_3);
VRT_re_fini(VGC_re_4);
}
const char *srcname[2] = {
"input",
"Default",
};
const char *srcbody[2] = {
/* "input"*/
"backend default {\n"
" .host = \"localhost\";\n"
" .port = \"80\";\n"
"}\n"
"\n"
"sub vcl_recv {\n"
" if (req.url ~ \"(\\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=\") {\n"
" set req.url = regsuball(req.url, \"&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\\-\\.%25]+)\", \"\");\n"
" set req.url = regsuball(req.url, \"\\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\\-\\.%25]+)\", \"?\");\n"
" set req.url = regsub(req.url, \"\\?&\", \"?\");\n"
" set req.url = regsub(req.url, \"\\?$\", \"\");\n"
" }\n"
" }\n"
"",
/* "Default"*/
"/*-\n"
" * Copyright (c) 2006 Verdens Gang AS\n"
" * Copyright (c) 2006-2009 Linpro AS\n"
" * All rights reserved.\n"
" *\n"
" * Author: Poul-Henning Kamp <phk@phk.freebsd.dk>\n"
" *\n"
" * Redistribution and use in source and binary forms, with or without\n"
" * modification, are permitted provided that the following conditions\n"
" * are met:\n"
" * 1. Redistributions of source code must retain the above copyright\n"
" * notice, this list of conditions and the following disclaimer.\n"
" * 2. Redistributions in binary form must reproduce the above copyright\n"
" * notice, this list of conditions and the following disclaimer in the\n"
" * documentation and/or other materials provided with the distribution.\n"
" *\n"
" * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND\n"
" * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n"
" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n"
" * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE\n"
" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n"
" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n"
" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR \n"
" * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n"
" * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE\n"
" * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,\n"
" * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n"
" *\n"
" * $Id$\n"
" *\n"
" * The default VCL code.\n"
" *\n"
" * NB! You do NOT need to copy & paste all of these functions into your\n"
" * own vcl code, if you do not provide a definition of one of these\n"
" * functions, the compiler will automatically fall back to the default\n"
" * code from this file.\n"
" *\n"
" * This code will be prefixed with a backend declaration built from the\n"
" * -b argument.\n"
" */\n"
"\n"
"sub vcl_recv {\n"
" if (req.restarts == 0) {\n"
"\tif (req.http.x-forwarded-for) {\n"
"\t set req.http.X-Forwarded-For =\n"
"\t\treq.http.X-Forwarded-For \", \" client.ip;\n"
"\t} else {\n"
"\t set req.http.X-Forwarded-For = client.ip;\n"
"\t}\n"
" }\n"
" if (req.request != \"GET\" &&\n"
" req.request != \"HEAD\" &&\n"
" req.request != \"PUT\" &&\n"
" req.request != \"POST\" &&\n"
" req.request != \"TRACE\" &&\n"
" req.request != \"OPTIONS\" &&\n"
" req.request != \"DELETE\") {\n"
" /* Non-RFC2616 or CONNECT which is weird. */\n"
" return (pipe);\n"
" }\n"
" if (req.request != \"GET\" && req.request != \"HEAD\") {\n"
" /* We only deal with GET and HEAD by default */\n"
" return (pass);\n"
" }\n"
" if (req.http.Authorization || req.http.Cookie) {\n"
" /* Not cacheable by default */\n"
" return (pass);\n"
" }\n"
" return (lookup);\n"
"}\n"
"\n"
"sub vcl_pipe {\n"
" # Note that only the first request to the backend will have\n"
" # X-Forwarded-For set. If you use X-Forwarded-For and want to\n"
" # have it set for all requests, make sure to have:\n"
" # set bereq.http.connection = \"close\";\n"
" # here. It is not set by default as it might break some broken web\n"
" # applications, like IIS with NTLM authentication.\n"
" return (pipe);\n"
"}\n"
"\n"
"sub vcl_pass {\n"
" return (pass);\n"
"}\n"
"\n"
"sub vcl_hash {\n"
" set req.hash += req.url;\n"
" if (req.http.host) {\n"
" set req.hash += req.http.host;\n"
" } else {\n"
" set req.hash += server.ip;\n"
" }\n"
" return (hash);\n"
"}\n"
"\n"
"sub vcl_hit {\n"
" if (!obj.cacheable) {\n"
" return (pass);\n"
" }\n"
" return (deliver);\n"
"}\n"
"\n"
"sub vcl_miss {\n"
" return (fetch);\n"
"}\n"
"\n"
"sub vcl_fetch {\n"
" if (!beresp.cacheable) {\n"
" return (pass);\n"
" }\n"
" if (beresp.http.Set-Cookie) {\n"
" return (pass);\n"
" }\n"
" return (deliver);\n"
"}\n"
"\n"
"sub vcl_deliver {\n"
" return (deliver);\n"
"}\n"
"\n"
"sub vcl_error {\n"
" set obj.http.Content-Type = \"text/html; charset=utf-8\";\n"
" synthetic {\"\n"
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n"
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\n"
" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n"
"<html>\n"
" <head>\n"
" <title>\"} obj.status \" \" obj.response {\"</title>\n"
" </head>\n"
" <body>\n"
" <h1>Error \"} obj.status \" \" obj.response {\"</h1>\n"
" <p>\"} obj.response {\"</p>\n"
" <h3>Guru Meditation:</h3>\n"
" <p>XID: \"} req.xid {\"</p>\n"
" <hr>\n"
" <p>Varnish cache server</p>\n"
" </body>\n"
"</html>\n"
"\"};\n"
" return (deliver);\n"
"}\n"
"",
};
static struct director *directors[2];
const struct VCL_conf VCL_conf = {
.magic = VCL_CONF_MAGIC,
.init_func = VGC_Init,
.fini_func = VGC_Fini,
.ndirector = 2,
.director = directors,
.ref = VGC_ref,
.nref = VGC_NREFS,
.nsrc = 2,
.srcname = srcname,
.srcbody = srcbody,
.recv_func = VGC_function_vcl_recv,
.pipe_func = VGC_function_vcl_pipe,
.pass_func = VGC_function_vcl_pass,
.hash_func = VGC_function_vcl_hash,
.miss_func = VGC_function_vcl_miss,
.hit_func = VGC_function_vcl_hit,
.fetch_func = VGC_function_vcl_fetch,
.deliver_func = VGC_function_vcl_deliver,
.error_func = VGC_function_vcl_error,
};
A lot longer than that VCL snippet, isn't it? The choice to use a custom language at the time was reasonable–it allows us to get some of the speed of C, while making it more difficult to make certain kinds of mistakes.
Varnish itself does provide an escape hatch that allows for writing inline C code in VCL. Fastly disallows use of this because it's quite unsafe, especially in a SaaS environment.
There are of course, tradeoffs here. It would be a huge problem if some VCL
accidentally goes into an infinite loop, so VCL doesn't allow any looping
constructs at all. No while or for loops, and the language prohibits use of
recursion. The data types available are meagre at best–we aren't going to be
creating any arrays and adding entries to them during request processing. While
Fastly extended VCL to change this, the original version of the language that
they forked from didn't even allow user-defined variables at all:
There are no user-defined variables; values can only be assigned to variables attached to backend, request or document objects.
Limitations like this are why it can take some creative coding backflips to pull off tasks like validating and parsing JWTs entirely in VCL. Some desired features may just not be possible to implement at all–for example, if a JWT is encrypted with an algorithm that isn't supported by the crypto functions exposed via Fastly's VCL extensions, then we're out of luck.
We'd probably not make this same design choice in 2024. We have all sorts of languages and virtual machines that it might make more sense to use, but we have the benefits of experience, hindsight, and some large shoulders to stand on.
We'll address this in more detail later, but this is where the Fastly edge compute product comes into play. Using that removes a lot of limitations, since we're able to write code that runs at the edge in any language that can be compiled down to WebAssembly.
VCL Pointers
This documentation isn't going to be a fully-fledged tutorial on VCL, because there's already a decent trove of such out there in the wild. Rather, we'll present some guidance on what to read to understand and work with VCL effectively.
The Varnish 2.1.5 VCL Reference
As mentioned earlier, Fastly diverged from the open-source Varnish distribution at around version 2.1.5. The VCL 2.1.5 reference documentation thus serves as a decent foundation in getting started with VCL.
Always keep in mind when reading this documentation that there are certain features not present in Varnish 2.1.5 that are available in Fastly VCL–for example, user-defined variables are available in Fastly VCL, but not Varnish 2.1.5 VCL.
It's best to think of VCL as what the docs say it is–a language that's superficially similar to C and Perl, but with many more constraints.
The Fastly VCL Documentation
Fastly have documentation available on both using VCL, and a reference on the language including their proprietary extensions.
They've provided quite a number of useful extensions in the form of both fundamental language changes and functions that can be invoked. For example, user-defined variables can be used in a subroutine in Fastly VCL like so:
declare local var.some_message STRING;
declare local var.some_count INTEGER;
set var.some_message = "hello world";
set var.some_count = 1;
Need to, for some reason, do some trigonometry in VCL? There's a whole slew of trig functions one can use. More likely, need to use some randomness in your code? There are also functions available for that.
I recommend at least reading the sections on Declarations, Operators, Statements, Subroutines, and Types top-to-bottom. The Function references can be read on an as-needed basis, though it doesn't hurt to familiarize yourself with the categories of functions available.
There is also a VCL best practices document that is a must-read.
Fastly Fiddle
Fastly Fiddle is a playground for testing VCL code, and it's immensely useful. Here's the one thing to really remember when working with it, in big, red, bold text.
Fiddles are public and there's nothing you can do to change that and make them private. Do not put anything you wouldn't want the public to see into a Fiddle. No production credentials. No secret endpoints. None of that kind of thing.
Okay. Got that? Great. Fastly Fiddle takes the VCL code thrown into the text areas, validates it, and then starts up a Fastly service with said VCL. By default it sets the origin to HTTP me!, a small Glitch (another Fastly product) application that acts as a sandbox for handling requests and serving up canned responses. The origin can be changed to point at something different, like a HTTPBin instance, or real live services running somewhere.
Fiddle can be a little slow sometimes, which is painful. Copying-and-pasting code around is also kind of tedious. Still, it's better than nothing, and being able to share and embed Fiddles with other folks is quite nice.
Falco
Falco comes with a "caveat emptor". It's a Fastly-specific VCL development tool, but it isn't actually built or maintained by Fastly itself. It's quite nice though, and it integrates with VSCode via Fastly's extension.
Of particular note is that it can be used on the command line as a REPL:
$ falco console
=========================================================
falco console tool - Inline evaluation for input string
=========================================================
Control Commands:
\s, \scope [scope] : Change running scope
\h, \help : Display help
\q, \quit : Quit from console
@RECV>> declare local var.foobar STRING;
@RECV>> set var.foobar = "Hello World";
@RECV>> var.foobar
(STRING)Hello World
@RECV>>
It also has testing support, and generally just provides development experience with shorter feedback loops than waiting for Fastly Fiddle to deploy a service.
The reason I can't wholeheartedly recommend it is that it's not an official Fastly project, and I have run into differences between its behaviour and actual Fastly VCL behaviour in the past. Regardless, it's quite promising, and is an actively developed project. I hope that we can shift more of our automated testing strategy over to using it rather than the full-VCL-deployment-to-Fastly-and-make-real-requests integration style testing we're using today.
Gotchas and Tips
No Subroutine Arguments
Want to define a custom subroutine that, say, adds two numbers?
sub add_two(a INTEGER, b INTEGER) INTEGER {
return a + b;
}
sub vcl_recv {
declare local var.result INTEGER;
set var.result = add_two(2, 2);
log "addition result: " + std.itoa(var.result);
}
Haha! You can't. Sorry. The above syntax isn't actually valid, subroutines cannot take arguments, but they can return values. Yes, this causes untold amounts of pain and weird workarounds.
Aside from subroutines not taking arguments, not even the + would work as
expected. From the Fastly VCL docs on operators:
Note that as there are currently no numeric expressions in general; these operators are limited to use with specific operands. For example, var.i < 5 is permitted but 2 < 5 is not.
You'll run into weird cases where like, 2 + 2 doesn't even work the way one
would expect, because + is for string concatenation.
A common workaround one will see is to essentially use the req.http variable
as a global map of STRINGs that can be referenced later,
as mentioned in the Fastly documentation:
Request headers are also frequently used as variables to store strings for later use.
However, the following works...
sub add_two INTEGER {
// Implicitly initialized to 0
declare local var.result INTEGER;
set var.result += std.strtol(req.http.operand-a, 10);
set var.result += std.strtol(req.http.operand-b, 10);
return var.result;
}
sub vcl_recv {
declare local var.result INTEGER;
set req.http.operand-a = "2";
set req.http.operand-b = "2";
set var.result = add_two();
log "addition result: " + std.itoa(var.result);
}
Welcome to hell.
This, as you can see, is tedious. For this reason one may make different
factoring decisions when writing VCL code than they would in other
languages–extraction of certain functionality into subroutines may be more
trouble than it's worth. Worse, if you're not careful and don't unset those
values on req.http before a request is actually made, they'll be sent as
headers.
Tread carefully here, and think about whether there's some kind of request or
response data that naturally makes sense to read and perhaps modify in-place
rather than trying to pass one-off values into subroutines via req.http.
This may not come naturally when one has internalized a tendency to see globals and side-effects as undesirable code smells in other, lets say more robust, languages.
Invoking Subroutines
If a user-defined subroutine does not return a value, e.g.
sub emit_log {
log "foo-bar"
}
Then it must be invoked via call, not using parentheses. e.g
call emit_log;
Logging
There are three things of extra-special note about logging in Fastly.
- Fastly charges for log emission, so a
log "foo"may incur costs. - The log parsers we have in place in our logging system expect a very specific log format anyway, and emitting random log messages would break that parsing.
log "foo"doesn't actually work outside of Fiddle the way you'd expect anyhow. The documentation on logging goes into further detail.
Generally, we log once, in vcl_log.
This section is a WIP and will refer to the correct templates to modify in the relevant repos in the future.
Variable Scope
Not all variables are available in all scopes. For example, bereq, the object
that encapsulates the backend request, is not available at all in the vcl_recv
scope. In vcl_recv we have no idea if the request will result in a cache HIT
or not, so why would it be?
The Fastly documentation contains a matrix that indicates which variables are readable and writable in which states.
Percent Escaping
This has tripped the author up before. URL-safe percent encodings like %20 are
actually unescaped to the corresponding character in VCL.
Be careful when copying and pasting strings into VCL.
Wait Wait Wait. Where Do I Put Custom VCL?
Work in progress.