English – cat /dev/random

Generate authorization header for Amazon S3 in Varnish

9月 062012

Amazon S3 REST-API is necessary to generate signature.
vmod-awsrest generate to Authorization and Date header for Amazon S3.

How to use

VCL


import awsrest;

backend default {
  .host = "s3.amazonaws.com";
  .port = "80";
}

sub vcl_recv{
  awsrest.s3_generic(
  "accessKey",            //AWSAccessKeyId
  "secretKey",            //SecretAccessKeyID
  req.request,            //HTTP-Verb
  req.http.content-md5,   //Content-MD5
  req.http.content-type,  //Content-Type
  "",                     //canonicalizedAmzHeaders
  req.url,                //canonicalizedResource
  now                     //Date
  );
}

Output


15 TxHeader     b Date: Tue, 03 Jul 2012 16:21:47 +0000
15 TxHeader     b Authorization: AWS accessKey:XUfSbQDuOWL24PTR1qavWSr6vjM=

This module set to req.http.Authorization and req.http.Date, bereq is not use.
I recommend call in the vcl_recv.
And, be careful to default settings.
If req.http.Authorization contains, it is not caching. (default setting)

download here.
libvmod-awsrest

—
Reference site
PHP で Amazon S3 の REST API を使用 #1
Authenticating REST Requests

日本語はこっち
VarnishでAmazon S3の認証ヘッダを作るVMODを作ってみた

はてなブックマーク - Generate authorization header for Amazon S3 in Varnish

To parsing the POST/GET request and Cookie header of Varnish (vmod-parsereq)

English, 技術 1 Response »

6月 132012

Difficult to access the POST/GET/Cookie value in Varnish.
I want more easily to access it.
Therefore, I tried to make a VMOD to parse.

Feature

Support POST/GET request and Cookie header.
Support application/x-www-form-urlencoded and multipart/form-data Content-type in POST request.
Can be getting a list of key.
Support Varnish 3.0.1, 3.0.2 , 3.0.3-rc1

How to use

For example, how to set the response header from the POST key of hoge.
It’s a simple.


import parsereq;

vcl_recv{
  //please write "parsereq.init();" to 1st line in vcl_recv.
  parsereq.init();
}
vcl_deliver{
  set resp.http.hoge = parsereq.post_header("hoge");
}

Future plans

urlencode/decode
change the value and set to bereq
refactoring

I hope that this code is of help to you.

download here.
vmod-parsereq

はてなブックマーク - To parsing the POST/GET request and Cookie header of Varnish (vmod-parsereq)

How to hook VCL function at VMOD

English, 技術 No Responses »

5月 262012

VMOD processing want to before/after VCL function processing.
But, I do not want to write extra line.
How to do it?

Can be realized by the replace the pointer to sp->vcl->XXXX_func (VCL_XXXX)

Can be added to the processing by replacing pointer



static vcl_func_f         *vmod_redirect_Hook_vcl_error = NULL;
static pthread_mutex_t    vmod_redirect_mutex = PTHREAD_MUTEX_INITIALIZER;
static unsigned           hook_done = 0;

static int vmod_Hook_vcl_error(struct sess *sp){

    ....    //write the processing

    //call original vcl_error
    return(vmod_redirect_Hook_vcl_error(sp));

}

int
vmod_location(struct sess *sp, int status, const char*p,...)
{
        //vcl reload detection.
        if(hook_done == 1
           && sp->vcl->error_func != vmod_redirect_Hook_vcl_error ) hook_done = 0;

        if(hook_done == 0){
            //lock to prevent another thread write
            AZ(pthread_mutex_lock(&vmod_redirect_mutex));

            if(hook_done == 0)
            {
                //store the original vcl_error pointer
                vmod_redirect_Hook_vcl_error = sp->vcl->error_func;

                //hook
                sp->vcl->error_func = vmod_Hook_vcl_error;

                hook_done = 1;
            }
            //unlock
            AZ(pthread_mutex_unlock(&vmod_redirect_mutex));
        }

    ....

    return (status);
}

(via:vmod_redirect)

Hook to vcl_error at vmod processing in first time.
because there is no way to write-access the pointer at vmod_Init.

also, has locked using the mutex, because to prevent the loop by thread concurrent access.

I hope that this code is of help to you.

ATTENTION
don’t use varnishadm’s command “vcl.use” and “vcl.discard” . because to the segfault or call to other vcl function.

—
modify(2012-08-01)
when you vcl reloaded, hook method be off.

はてなブックマーク - How to hook VCL function at VMOD

Perform an ACL match to X-Forwarded-For (varnish)

English, 技術 No Responses »

4月 132012

How do ACL match to req.http.X-Forwarded-For? (this is string!!)
I tried to make a vmod.

inet_pton(BOOL ipv6 , STRING str , STRING defaultstr)

example 1


import campur_xcir;

set resp.http.v6 = campur_xcir.inet_pton(true,"2001:0db8:bd05:01d2:288a:1fc0:0001:10ee","1982:db8:20:3:1000:100:20:3");
set resp.http.v4 = campur_xcir.inet_pton(false,"1.1.1.1","2.2.2.2");
set resp.http.v6ng = campur_xcir.inet_pton(true,"2001:0db8:bd05:01d2:288a:1fc0:0001:10eeHOGE","1982:db8:20:3:1000:100:20:3");//NG pattern
set resp.http.v4ng = campur_xcir.inet_pton(false,"1.1.1.1HOGE","2.2.2.2");//NG pattern

//result
v6: 2001:db8:bd05:1d2:288a:1fc0:1:10ee
v4: 1.1.1.1
v6ng: 1982:db8:20:3:1000:100:20:3
v4ng: 2.2.2.2

example 2


import campur_xcir;

//acl
acl local {
    "192.168.1.0"/24;
    !"0.0.0.0";
}

sub vcl_recv{
  if(campur_xcir.inet_pton(false , req.http.X-Forwarded-For , "0.0.0.0") ~ local){
      //acl ok
      ...
  }
}

I hope that this code is of help to you.
libvmod-campur_xcir

this module is my motley function.
Others, get varnish generated hash etc…

this vmod’s function is increase at times. 🙂

はてなブックマーク - Perform an ACL match to X-Forwarded-For (varnish)

[tips]varnishncsa save the split log for every host

English, 技術 1 Response »

3月 202012

To save the split log for every host([hostname].access_log), Apache is easy.
I want the same action in varnishncsa. What should I do?

Use the options -m [tag:regex], -w [file], -a and -D.

-m perform a regex match to the tag’s log entry.
-w write log to a file.
-a append log. Will be overwritten if you do not specify.
-D Daemonize.

exec varnishncsa(host is a.example.net and b.example.net)


varnishncsa -m "RxHeader:^Host: a.example.net$" -a -w /var/log/varnish/a.example.net.access_log -D
varnishncsa -m "RxHeader:^Host: b.example.net$" -a -w /var/log/varnish/b.example.net.access_log -D

after request


cat a.example.net.access_log
192.168.1.199 - - [20/Mar/2012:12:51:50 +0900] "GET http://a.example.net/a HTTP/1.0" 200 280 "-" "Wget/1.12 (linux-gnu)"

cat b.example.net.access_log
192.168.1.199 - - [20/Mar/2012:12:51:59 +0900] "GET http://b.example.net/a HTTP/1.0" 200 280 "-" "Wget/1.12 (linux-gnu)"

work as expected.

If you want to log rotate, please send the SIGHUP.

はてなブックマーク - [tips]varnishncsa save the split log for every host

varnishlogtrans version up(mapping from vcl_trace to vcl source)

English, 技術 No Responses »

3月 122012

varnishlogtrans version up(mapping from vcl_trace to vcl source)

Past Articles:convert varnishlog output to easy

Changes

Feature: mapping from vcl_trace to vcl source(line).(request from @perbu)

Feature: show backend status.(request from @dai_yamashita)

Change: Add run option.

Experimental: mapping from vcl_trace to vcl source(all).

mapping from vcl_trace to vcl source(line).

if you set to -f option, replace vcl_trace to VCL code
before.


         | trace  | vrt_count:1 vcl_line:55 vcl_pos:1

after(use -f and -e option)


         | trace  | set req.http.X-TEST1=campur_xcir.gethash();
         |        | ^ (vrt_count:1 vcl_line:55 vcl_pos:1 src_name:input)

show backend status.

show backend status.
past status is window 4times.


backend status
------------------------------------------------------------

name    | status  | past status
default | healthy | YY

Add run option.

-f [vcl file]

if you set, you can mapping from vcl_trace to vcl source.

-e [varnishd]

if you set, you can mapping from vcl_trace to vcl source.

-cc_command [cc_command]

if you set in the -f option and varnish’s cc_command option ,please setting.

–(backend|action|variable) [on|off]

if you want to disable (backend|action|variable) info , set to off. (default is on)

mapping from vcl_trace to vcl source(all).

this function is unstable.
if you want use this function.
it need change value and set option(–src=on -f=[vclfile] -e=[varnishd]).

before


define('experimental' , false);

after


define('experimental' , true);

output


############################################################
vcl trace
[exec]      /* "input"*/
[exec]  /*

...

[exec]  sub vcl_recv {
[exec]      if (req.restarts == 0) {
[exec]          if (req.http.x-forwarded-for) {
                    set req.http.X-Forwarded-For =
                        req.http.X-Forwarded-For + ", " + client.ip;
[exec]          } else {
[exec]              set req.http.X-Forwarded-For = client.ip;
[exec]          }
[exec]      }
[exec]      if (req.request != "GET" &amp;amp;amp;&amp;amp;amp;
[exec]        req.request != "HEAD" &amp;amp;amp;&amp;amp;amp;
[exec]        req.request != "PUT" &amp;amp;amp;&amp;amp;amp;
[exec]        req.request != "POST" &amp;amp;amp;&amp;amp;amp;

...

I hope that this code is of help to you.

download here.
varnishlogTrans

はてなブックマーク - varnishlogtrans version up(mapping from vcl_trace to vcl source)

convert varnishlog output to easy

English, 技術 No Responses »

2月 092012

Do you use a Varnish? I’ve been using.
I think to using varnishlog for VCL debugging.

For example, the output of the varnishlog that include ESI request.

varnishlog(include ESI)


   13 BackendOpen  b default 192.168.1.199 36011 192.168.1.199 81  ★Open backend here.
   13 TxRequest    b GET
   13 TxURL        b /test2.html
   13 TxProtocol   b HTTP/1.1
   13 TxHeader     b User-Agent: Wget/1.12 (linux-gnu)
   13 TxHeader     b Accept: */*
   13 TxHeader     b Host: 192.168.1.199:6081
   13 TxHeader     b X-Varnish: 171551254
   13 TxHeader     b Accept-Encoding: gzip
   13 RxProtocol   b HTTP/1.1
   13 RxStatus     b 200
   13 RxResponse   b OK
   13 RxHeader     b Date: Sat, 28 Jan 2012 05:05:59 GMT
   13 RxHeader     b Server: Apache/2.2.15 (Scientific Linux)
   13 RxHeader     b Last-Modified: Sat, 23 Jul 2011 08:09:05 GMT
   13 RxHeader     b ETag: "5a28-51-4a8b8187f8177"
   13 RxHeader     b Accept-Ranges: bytes
   13 RxHeader     b Content-Length: 81
   13 RxHeader     b Connection: close
   13 RxHeader     b Content-Type: text/html; charset=UTF-8
   13 Fetch_Body   b 4(length) cls 0 mklen 1
   13 Length       b 81
   13 BackendClose b default  ★Close backend here.
   13 BackendOpen  b default 192.168.1.199 36012 192.168.1.199 81  ★Open backend for ESI request here.
   13 TxRequest    b GET
   13 TxURL        b /test3.html
   13 TxProtocol   b HTTP/1.1
   13 TxHeader     b User-Agent: Wget/1.12 (linux-gnu)
   13 TxHeader     b Accept: */*
   13 TxHeader     b Host: 192.168.1.199:6081
   13 TxHeader     b X-Varnish: 171551254
   13 TxHeader     b Accept-Encoding: gzip
   13 RxProtocol   b HTTP/1.1
   13 RxStatus     b 200
   13 RxResponse   b OK
   13 RxHeader     b Date: Sat, 28 Jan 2012 05:05:59 GMT
   13 RxHeader     b Server: Apache/2.2.15 (Scientific Linux)
   13 RxHeader     b Last-Modified: Sat, 23 Jul 2011 08:08:59 GMT
   13 RxHeader     b ETag: "5a29-14-4a8b81829c835"
   13 RxHeader     b Accept-Ranges: bytes
   13 RxHeader     b Content-Length: 20
   13 RxHeader     b Connection: close
   13 RxHeader     b Content-Type: text/html; charset=UTF-8
   13 Fetch_Body   b 4(length) cls 0 mklen 1
   13 Length       b 20
   13 BackendClose b default  ★Close backend here.
   12 SessionOpen  c 192.168.1.199 55115 :6081
   12 ReqStart     c 192.168.1.199 55115 171551254 ★Start Request (/test2.html)
   12 RxRequest    c GET
   12 RxURL        c /test2.html
   12 RxProtocol   c HTTP/1.0
   12 RxHeader     c User-Agent: Wget/1.12 (linux-gnu)
   12 RxHeader     c Accept: */*
   12 RxHeader     c Host: 192.168.1.199:6081
   12 RxHeader     c Connection: Keep-Alive
   12 VCL_call     c recv 1 7.1
   12 VCL_Log      c a
   12 VCL_trace    c 2 9.1
   12 VCL_Log      c b
   12 VCL_trace    c 3 11.1
   12 VCL_Log      c a
   12 VCL_return   c lookup
   12 VCL_call     c hash 18 85.5
   12 Hash         c /test2.html
   12 VCL_trace    c 19 87.9
   12 Hash         c 192.168.1.199:6081
   12 VCL_return   c hash
   12 VCL_call     c miss 22 99.5 fetch
   12 Backend      c 13 default default　★Call backend.
   12 TTL          c 171551254 RFC 120 -1 -1 1327727160 0 1327727159 0 0
   12 VCL_call     c fetch 4 18.1 deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Sat, 28 Jan 2012 05:05:59 GMT
   12 ObjHeader    c Server: Apache/2.2.15 (Scientific Linux)
   12 ObjHeader    c Last-Modified: Sat, 23 Jul 2011 08:09:05 GMT
   12 ObjHeader    c ETag: "5a28-51-4a8b8187f8177"
   12 ObjHeader    c Content-Type: text/html; charset=UTF-8
   12 VCL_call     c deliver 26 116.5 deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.15 (Scientific Linux)
   12 TxHeader     c Last-Modified: Sat, 23 Jul 2011 08:09:05 GMT
   12 TxHeader     c ETag: "5a28-51-4a8b8187f8177"
   12 TxHeader     c Content-Type: text/html; charset=UTF-8
   12 TxHeader     c Date: Sat, 28 Jan 2012 05:05:59 GMT
   12 TxHeader     c X-Varnish: 171551254
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: close
   12 VCL_call     c recv 1 7.1  ★Start ESI log.
   12 VCL_Log      c a
   12 VCL_trace    c 2 9.1
   12 VCL_Log      c b
   12 VCL_trace    c 3 11.1
   12 VCL_Log      c a
   12 VCL_return   c lookup
   12 VCL_call     c hash 18 85.5
   12 Hash         c /test3.html
   12 VCL_trace    c 19 87.9
   12 Hash         c 192.168.1.199:6081
   12 VCL_return   c hash
   12 VCL_call     c miss 22 99.5 fetch
   12 Backend      c 13 default default
   12 TTL          c 171551254 RFC 120 -1 -1 1327727160 0 1327727159 0 0
   12 VCL_call     c fetch 4 18.1 deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Sat, 28 Jan 2012 05:05:59 GMT
   12 ObjHeader    c Server: Apache/2.2.15 (Scientific Linux)
   12 ObjHeader    c Last-Modified: Sat, 23 Jul 2011 08:08:59 GMT
   12 ObjHeader    c ETag: "5a29-14-4a8b81829c835"
   12 ObjHeader    c Content-Type: text/html; charset=UTF-8
   12 VCL_call     c deliver 26 116.5 deliver
   12 Length       c 20
   12 ReqEnd       c 171551254 1327727159.889132023 1327727159.890880346 -0.001736164 nan nan  ★Request end.
   12 SessionClose c EOF mode
   12 StatSess     c 192.168.1.199 55115 0 1 1 0 0 2 282 20
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1327727160 1.0

little too difficult for me.
Therefore, I tried to make a tool to format and display.
Continue reading »

はてなブックマーク - convert varnishlog output to easy

Varnish redirect operation to easy(libvmod-redirect)

English, 技術 No Responses »

2月 052012

Varnish redirect is pain.


sub vcl_recv {
  if (req.http.user-agent ~ "iP(hone|od)") {
    error 750 "Moved Temporarily";
  }
}

sub vcl_error {
  if (obj.status == 750) {
    set obj.http.Location = "http://www.example.com/iphoneversion/";
    set obj.status = 302;
    return(deliver);
  }
}

(via:Redirecting using VCL)
I just want to redirect. but should write code in two action.
This is it lack of maintenance, and I do not like.
I tried to make a vmod_redirect can be easily redirect.
Continue reading »

はてなブックマーク - Varnish redirect operation to easy(libvmod-redirect)

Varnish3’s transient storage size control

English, 技術 No Responses »

7月 132011

If you manage many objects with high traffic , Varnish get a swap (although it rare)

Varnish 3 have new parameter “shortlived”.
This parameter description is

Objects created with TTL shorter than this are always put in transient storage.

transient storage is malloc storage.



storage.list
200
Storage devices:
        storage.Transient =*** malloc ***
        storage.s0 = malloc

はてなブックマーク - Varnish3’s transient storage size control

VCL and parameter Changes in Varnish Cache 3.0

English, 技術 No Responses »

7月 082011

Varnish Cache 3.0 has a lot of incompatible changes.
I tried summarize the incompatible changes.

I’m not good at English.
VCL and parameter Changes in Varnish Cache 3.0 (PDF)

—–
Version 2.0 to 2.1
see:
incompatible changes in Varnish 2.1 (Japanese)

はてなブックマーク - VCL and parameter Changes in Varnish Cache 3.0