We all want to reach the top of Google & other search engines with the help of SEO: Because organic conversion is free, and users trust organic search results more than ads. After all, they’ve found the result themselves.

However, writing for search engines and in line with SEO rules is not an easy task. Writers used to only focus on content, but now we also have to take technical SEO requirements into account and focus on a keyword and supporting keywords. A challenging process. Our six tips show how to write search enginge optimized copy that will make your content stand above the crowd.

SEO advice 1: first content, then keywords

If you try and fit as many SEO keywords into a text as possible, it soon starts sounding quite stiff. So we recommend going ‘old school’ when writing a text: introduction, main section, conclusion. Make sure that your text is tailored to the reading habits of web users. Once the text is finished, it’s time to weave in your SEO keyword and the supporting keywords. SEO writing therefore comes at the end, another step in the process.

SEO advice 2: use synonyms

Thankfully, our colleagues at Google & other search engines are clever engineers, and are making their search engines increasingly smart. Their algorithms can now recognise synonyms and contexts. This means that you can use alternative words and even long-winded descriptions for your keyword and supporting keywords when getting your text SEO ready.

SEO advice 3: play with formats

Who reads all the way to the end of a text nowadays? A long article might make sense for specialist topics, but our experience is that people look for entertainment. Since search engines love what’s popular, text only is no good idea (retention time and page views per session are key concepts). Here’s a few formatting ideas to stop your SEO from becoming too boring.

• Advice article (like this one)
• Photos with captions
• Infographic with written explanation
• Interview
• Article with ‘decorative materials’ like titles, subtitles, quotations and info boxes
• Q&A article (buzzword: featured snippets in search engines)

Whatever presentation you choose, test the mobile view as well. Make sure that your SEO content is still easy and enjoyable to read on mobile devices.

SEO advice 4: dip into your bag of tricks

What every SEO content manual says but is often forgotten: body text isn’t the only place you can put SEO keywords. Make sure that the keyword is in the URL, title tag, all image and video names, and above all in the heading (‘H1’). This means that your keyword has already appeared at least five times without overdoing the text. However, with all these technical tricks, don’t forget the copy itself – so make sure that the SEO keyword appears particularly frequently in the first few paragraphs of text. Google & other search engines will love you for it.

SEO advice 5: do the maths

As a rule of thumb, an SEO keyword should make up 3-5% of the text. There are various online tools you can use to check keyword density, this is one we like.

The statistics for this text are the following
Main keyword: ‘SEO’ used 34 times = 4%
Supporting keyword: ‘keyword’ used 17 times = 2%

However, this is quite a long text. A 300-word text is enough for a search engine to work with. This means that the keyword needs to appear about 15 times. Instead of using a calculation tool we recommend just highlighting the keywords in the text and then counting them up – this means that you keep an overview, and you know how much redrafting you need to do to keep the search engine happy.

SEO advice 6: don’t forget the translation

Give your translation company the right SEO keywords – because SEO keywords don’t come from dictionaries, they come from SEO specialists. And don't forget to keep an eye on what the translation company is doing, since translators are not SEO specialists. Make sure that the translator sticks as closely to your guidelines (and these tips) as the original text does.

Learnings

The good news for all fans of relevant, exciting content: high-quality material wins the day. Google clearly emphasised this by

integrating a quality algorithm into the search engine’s core algorithm with Panda. And that’s positive news because it rewards good work and not only SEO technicalities. However - good performance is much more than just SEO writing. If you want to know what else it takes, you best talk to this person.

Checklist

• SEO advice 1: content first, then keywords
  Have you written your text? Now you can add in your keyword to work on your SEO.

• SEO advice 2: use synonyms
  Write the ‘right’ text and use paraphrases for your keyword.

• SEO advice 3: play with formats
  Use formats that suit your users (and your keyword) so they truly engage with your content.

• SEO advice 4: dip into your bag of tricks
  Weave your keyword in everywhere – URL, title tag, H1-Hn, alt text, image names, image captions.

• SEO advice 5: do the maths
  4-5% keyword density is perfect for search engines.

• SEO advice 6: don’t forget the translation
  Give your translation company your keyword and supporting keywords– they won’t automatically know them.

The experts behind this article

Thanks to Fabian, Christoph, Jenny and Benoît for content and copy cleverness, and to Jérémie for the visual. This article would not have been possible without you!

One important task for such a project is to monitor the performance of the application. We usually use New Relic for that, but there's no New Relic for HHVM (yet), so we had to look into other solutions.

Since varnish is the entry point for all requests, be it PHP or HHVM or another backend, we searched for possibilities there. We found 2 vmods, libvmod-timers and libvmod-statsd, which make that possible.

Collecting response time and number of requests

It needs a statsd and a backend for collecting the data. We decided to test librato as the backend as we have heard good things about it (btw, if you have to configure many metrics in librato with the same attributes all over, use the API of librato, it's more powerful than their webfrontend and you can script it for later new metrics)

This setup allows us to collect detailed data about the performance of the different backends (and also monitor for example if there are too many 500 answers)

Compared to the examples in the blog posts about the vmods, I made some small changes, to get the correct data for us.

To be sure our API is “always” up, we use the fallback director of varnish, especially for the HHVM part (we have much more experience in running a PHP server than a HHVM, that's why). It looks like the following

director hhvm_api_fallback fallback {
  { .backend = hhvm_api; }
  { .backend = php_api; } // will only be used if hhvm_api is unhealthy.
}

This means, that varnish will normally use the hhvm_api backend, but if that is down, it will use the php_api backend. They run on the same server, but listen on different ports (of course, the whole thing is loadbalanced in front of it to have less single point of failures)

As we only route some requests to the hhvm_api, we have the following in vcl_recv:

if (req.url ~ "^/things.json?.*search=" ) {
    set req.backend = hhvm_api_fallback;
}

We can then later extend that URL matcher to include more requests.

The example of the author of the vmods uses req.http.X-Request-Backend for reporting which backend was used with statsd. Since with the fallback director defined above that is always _hhvm_apifallback, no matter which backend was used and we wanted to know when the fallback uses php_api instead of hhvm_api, we added the following to vcl_fetch:

set req.http.X-Stats-Backend = beresp.backend.name;

Now the backend that is actually used is reported later (at least for misses, for hits it's still hhvm_api_fallback, but that is ok, since it didn't use any backend at all, wthough I'm sure this could be changed as well somehow with some header trickery in the vcl).

The librato backend seems also to interpret some stats differently than for example the default graphite backend in statsd. Therefore we used the the following lines in vcl_deliver for sending the values to statsd:

### Key to use for statsd. Something like: config.hit.200
set req.http.X-Stats-Key =
    req.http.X-Stats-Backend + "." +
    req.http.X-Stats-HitMiss + "." +
    req.http.X-Stats-Status;

### Increment the amount of requests here, and how long this 
### particular backend took.
statsd.incr(   req.http.X-Stats-Key + ".count");
statsd.timing( req.http.X-Stats-Key + ".time", timers.req_response_time() );

We basically added .count and .time to the key as librato gets confuesd if you use the same key for counters and timers.

With this data we can now plot graphs for each backend and each request type in librato. We immediatly see, if there are problems (for example too many non-200 responses or if one backend is slower than expected). We also can compare hits vs misses and collect the load of the servers with collectd.

Logging slow requests

Having an overview about how fast your setup is, is one thing, but being able to know exactly which requests are slow is another one. For the PHP backend we have New Relic for that. It keeps track of slow requests with full call traces and eg. SQL queries and their performance. It's very useful, also for error reporting. For HHVM nothing similar exists yet (to our knowledge at least) and we didn't want to clutter the code with too many home-grown analysis calls. We decided that for now it's enough to just know which calls were slow, to be able to debug them, if it happens too often. And this is possible with varnish and the timers vmod. All we added for this in vcl_deliver was

std.log("Backend:" + req.http.X-Stats-Backend);
## Tag SlowQueries in the Log
if (timers.req_response_time() > 500) {
    std.log("SlowQuery:" + timers.req_response_time());
}

This always logs the used backend to the varnish logs. And if the response time was slower than 500ms, we also log it as SlowQuery to the logs.

Then we can use varnishncsa to get readable logs and write them to a file. We use the following command for that:

varnishncsa -F '%t "%r" %s %b "%{Referer}i" "%{User-agent}i" %{Varnish:hitmiss}x %{VCL_Log:Backend}x %D ' -m "VCL_Log:SlowQuery" -m "VCL_Log:Backend:hhvm_api$" -D -a -w logs/hhvm-api-slow.log

This is the usual logfile format of web servers with some info added at the end. If it was a hit or a miss, which backend was used (logged via the std.log command in the vcl) and how long it took (the %D parameter). Then we can filter to just log requests which have the SlowQuery tag and are for the hhvm_api backend (the others are recorded by New Relic, but we could of course do them here as well).

We then collect those with logstash and send them to our graylog2 server. From there we can do for example alerts or just analyse the requests and try to reproduce the slowness and make it faster.

It's of course not as powerful as the full toolset of New Relic (or similar), but at least we know now, which requests are slow. And we can also use that for other backends, should the need arise.

If you have any input what could be improved or if you know about other tools to collect stats for HHVM (or other currently exotic systems), let us know. We're eager to hear about them. (Btw, Javascript, browser-based solutions are out of question here, since the API backend just sends JSON or XML, no possibility to inject JavaScript).

“It scales”

I was mostly interested to know how ElasticSearch scales and performs under a very high number of queries. I'll try to summarize the findings below.

ElasticSearch is a fast and scalable, distributed search engine based on Lucene. One can define indexes that are horizontally split into shards. Shards are then automatically distributed by ElasticSearch on different nodes and can be replicated to be resilient in case of failure (using replicas). An ElasticSearch node is actually only a java process and can be run anywhere (nothing prevents you from having 2 nodes running on the same machine). Nodes discover themselves automatically via IP multicast or unicast to form an ElasticSearch cluster.

Example of a cluster with 2 nodes holding one index with 4 shards and 1 replica shard

In theory, it scales in both data capacity and number of queries simply by adding nodes to the cluster. When a new node is added to a cluster, ElasticSearch will automatically reassign some shards to the new node, reducing the load on existing node by putting it on the new one. All this happens automatically. An ElasticSearch node already takes advantage of multi-core CPUs so there isn't much to do there.

Node Roles Load Balancing

An ES node can play any, two or all of the 3 following roles:

• Master: master nodes are potential candidates for being elected master of a cluster. A cluster master holds the cluster state and handles the distribution of shards in the cluster. In fact, it takes care of the well-being of the cluster. When a cluster master goes down, the cluster automatically starts a new master election process and elects a new master from all the nodes with the master role.
• Data: data nodes hold the actual data in one or more shards, which are actually Lucene indexes. They are responsible for performing indexing and executing search queries.
• Client: client nodes respond to the ElasticSearch REST interface and are responsible for routing the queries to the data nodes holding the relevant shards and for aggregating the results from the individual shards.

By default, ElasticSearch nodes are configured to play the 3 roles, master, data and client. In an ElasticSearch cluster, there is usually more than one client node, and as they all offer a REST API, the client nodes can actually be used as load balancers, which eliminates the need of using an external load balancing mechanism. When a request is sent to an ElasticSearch cluster, the client node that receives the request always is the one that aggregates the results (that is never re-routed to another client node).

A proposed architecture do properly load balance requests from a web application (as we have in our project) is therefore the following:

Example of 2 application servers using ElasticSearch data nodes by talking to an embedded client on the same server

It is also recommended to keep a long live HTTP requests to the client nodes, as it will spare creation and deletion of a connection for each request.

Fast Data Nodes

As the data nodes hold the Lucene indexes, they should be configured to take the biggest advantage of Lucene's capabilities. I've summarized below the biggest learnings from the training:

“Make sure the file system cache is used by the ElasticSearch process”

As Lucene relies a lot on the file system cache for performance, it is highly recommended to leave at least half the memory free for the file system cache (make sure ES_HEAP_SIZE doesn't go above half the available memory). This also means that there should not be a bunch of other processes fighting for file system cache resources (e.g. a MySQL database). The more processes the more chances a file read will go to the disk and therefore slow the Lucene operations down.

“Make sure the process does not swap.” (cf. ElasticSearch reference guide)

The disk should only be used to read and write index files, if the process also uses it to swap the memory, the performance will degrade. A way to do that in Linux is to set bootstrap.mlockall to true.

“Watch the java pointers”

That one has to do with using Java in general. On a 64bit system, the ordinary object pointers (pop) can take up to 8 bytes of memory to be able to address the whole memory. However, the pointers can be compressed to take up only 4 bytes when the heap size is set to under 32gb. This is what the official Oracle doc says:

Compressed oops is supported and enabled by default in Java SE 6u23 and later. In Java SE 7, use of
compressed oops is the default for 64-bit JVM processes when -Xmx isn't specified and for values of
-Xmx less than 32 gigabytes. For JDK 6 before the 6u23 release, use the -XX:+UseCompressedOops
flag with the java command to enable the feature.

Therefore if ElasticSearch runs on a JDK before the 6u23 release, make sure you use -XX:UseCompressedOops flag if your heap size is less than 32gb.

“Files, files, files” (cf. ElasticSearch reference guide)

The ElasticSearch process can use up all file descriptors in the system. If this is the case make sure the file descriptor limit is set high enough for the user running the ElasticSearch process. Setting it to 32k or even 64k is recommended (you'll have to Google how to do that one …)

“Disk, disk, disk”

Medium and big hardware make more economical sense. Because of the way Lucene uses files (elaborate), SSDs are particularly suited, it is recommended to use RAID for performance only, and take advantage of ElasticSearch replicas for fault tolerance.

“Memory, memory, memory”

The optimal RAM size for a data node can be calculated by taking the total of all index files.

“Don't use facets on analyzed fields”

Facets on analyzed fields take a lot of memory. If the OutOfMemory error occurs, increase heap size or add more nodes…

Master Nodes, Shards, Replicas and the Cluster State

ElasticSearch indexes are split horizontally in shards and each shard is replicated in a configurable number of replicas. By default, the number of shards is set to 5 and the number of replica to 1, meaning an index will have 5 primary shards and 5 replica shards (1 replica for each primary shard). What this means is: 1 node is allowed to fail and the cluster will still contain the entire index (or: if more than 1 node fails, it is not guaranteed that the whole index is still present). If the number of replica is set to 2, then a maximum of 2 nodes can fail at the same time without any impact on data integrity (of course the cluster has to have more than 2 nodes in total…)

The ElasticSearch master is the one that is responsible to keep the integrity and status of the cluster. When a node is added to the cluster, it makes sure some shards are copied over to that node as soon as possible. When a node disappears from the cluster, it makes sure that missing shards are replicated on the existing nodes.

Nodes discover themselves using either multicast or unicast. It is highly recommended to use unicast instead of multicast for node discovery. Each node doesn't need to know about all other nodes of the cluster, since once a new node discovers another node, it will ask the master to provide the information about the whole cluster to the new node directly.

One (or two) more thing(s) …

Both the number of shards and the number of replicas can be configured independently for each index. The number of replicas can be changed at runtime, but the number of primary shards is defined when the index is created and cannot be changed, which raises the question: “How do I set the optimal number of shards for my application ?”. Well, there is no answer and that's in my opinion (a lot of participants decided to agree on that) the biggest limitation of ElasticSearch at the moment. Resizing the shards when the cluster is in operation is a tedious process, one way to do it is to setup a new cluster, create an index with the desired number of shards and copy the old index over to the new one. That operation is lengthy and no writes can be allowed when copying the data (which makes it not really realistic for big, rapidly changing indexes).

One alternative is to set the number of shards to a very high number. That is not recommended since each shard takes quite some overhead to maintain a Lucene index and, the smaller the shards, the bigger the overhead. They gave us some sort of formula to calculate the maximum shard size, and based on that, you can infer the number of shards you need if you know the size of your index (number_of_shards = index_size / max_shard_size). Here it is:

• Use one shard and start loading it with data
• When queries start to slow down and become too slow for you, the max capacity on that hardware is reached.

Another feature that they are working on is the possibility to easily backup and restore an index. That hopefully will help better manage a cluster. The technology is still young so it is understandable that the management capabilities are not yet up to the task compared with established solutions.

Memory on Client Nodes

One of the focus of the future ElasticSearch releases is to improve the memory consumption, above all on the client node. The client node is responsible for gathering the responses of the data nodes and for sorting among others. Those operations happen entirely in memory for now, which can lead to memory leaks if the entire result set doesn't fit there. The solution in that case is to increase the heap size of the client nodes. Future ElasticSearch versions will also try to use files take advantage of the file system cache, just like Lucene does.

Summary

ElasticSearch seems like a good, scalable technology and I'm looking forward to seeing it in full use with the current project. It seems like most of the things we learned at the training regarding performance and live operations are not really officially documented, which makes the training very valuable. The lessons from the trainings are the following:

• Java Heap should be big, but not too big
• Client nodes can be used as load balancers
• Watch out for the number of shards

Keep reading for more infos as soon as the project goes live !

The conference runs for a full day (8 hours worth of presentations) and can be viewed live or later on. We didn't manage to stay up till the end of it as the conference was schedule for happening during American day time, which more or less translate to European evening.

Anyway we have already watched a good part of it and another viewing session is about to be schedule soon for those who couldn't attend it.

In general the conference was interesting and the talkers made a good job at presenting their content. I personally found the following talks of particular interest.

Achieving Better Image Optimization by Billy Hoffman

This talk is about how you can gain performance by properly handling you image resources. Billy goes in every detail to explain us which file format to choose and what compression options are available. In most case he claims that you can reduce your images size by a significant percentage without losing any [viewable] quality.

It is known since some time that compressing your website's assets (js, css, html) is a good thing to do in order to make it faster. What Billy is pointing at, is that, although it is good to compress your css and js and you should still doing it, it would be much more efficient to start compressing your images as they usually represent the biggest part of the bandwidth used to transmit data from your webserver to your the web clients. Usually images account for about 75% of your page weight so reducing this amount will have a bigger impact than reducing the 25% page weight of the other assets.

Then a good part of the talk is dedicated to how to reduce your image size while keeping as much quality as possible. I would recommend any of you to have a look at this talk or at the slides. Even if you cannot remember all details just being aware of those concerns will be of great help to make your site faster.

Improving Request and Response Headers by Kyle Simpson

In this talk Kyle Simpson talks about all those request/response headers that are of little value and can be sometime a waste of bandwidth. Of course anyone would argue that headers are usually small enough to be insignificant compared to the html and images of a web page. But Kyle points out that those extra bytes sent and received can represent a large bandwidth consumption when we are talking about a website with a large audience and many thousand of HTTP requests per day.

One of the biggest waste is due to cookies. This header can get [relatively] big on many website (on average around 200 and 400 bytes). Although every one knows the usefulness of cookie, it is rarely acknowledge that they are useless in 80%, if not more, of your requests. Can anyone tell me what use of cookie can be done when getting a css file or an jpeg image? Bear in mind that the cookies are also sent back with the response, doubling the uselessness of those couple of hundred bytes in such cases.

The solution to this is simply to have your static assets served from another domain than your website. Being on another domain the browser won't send those cookie information and you will save some bandwidth and increase your website speed.

You can have a look here to find a written down version of what Kyle talked about in this talk.

Performance in Business Terms by Joshua Bixby

This talk from Joshua Bixby somehow echoed the keynote I have seen at Phpday 2011 by Steve Souders. In both of them, one of the main argument is that performance is your website's number 1 feature. This is almost contradictory to the popular believe that speed is not important as long as you have a long list of feature.

Well apparently this is wrong. Speed is important and to such an extent that the revenue you generate from your website depends on it. Many studies done by different website and organization (including amazon.com) found a strong correlation between your website's speed and the cash it generates. This is mostly true for merchant website but not only. To make it more general: if your website is fast then users will have pleasure to use and so will use it more. And the more they use it, the better it is. This will trigger more sells, more ad print, and more widespread awareness of your website and so on. So all in all it is good whatever your site business is.

To wrap up I believe that we should get more awareness about those simple things that can make your website faster as we all benefit from it, our users are happier, we spare on resources (bandwidth, storage, electrical) and we have a chance to increase our business (in the most general sense of it).