How do I fix my S3 / memcache latency issue?
We have SEVERE latency issues on our server.
We save 3 things of interest to S3 and add them to memcache.
- Custom avatars on average ~ 25k
- text ~ 1.5k
- xml ~ 1.5k
we have allocated 128meg RAM for memcached as it is now ... at the moment it spins at 74 megabytes
by doing some basic math we can easily have about 30,000 text documents (with their xml representations) and 1,000 custom avatars and are still under our 128 megs meant for memcache
right now we have ~ 100 custom avatars that could be pulled at any given time we have a hundred thousand text / xml documents but they are not viewable like avatars ... this one here, there type of thing
sometimes during the day, custom avatars load super slow (indicating that they should be loaded from s3) and at other times (after loading, of course) you can say that they are served with memcached; the same with text documents
we run merb under apache phusion with REE. we are using the evan weaver memcached gem built on libmemcached-0.25.14 (which I fully understand is not the most up to date library, this gem requires it)
From what I can see our latency issues are due to S3, which has severe latency issues (sometimes 500ms for a single avatar). However, it looks like this shouldn't be a problem as it should be cached all the time. By default, the cache is set to 1 week.
Relevant code:
@cache = MMCACHE.clone
begin
picture = @cache.get("/avatars/#{user.avatar}")
rescue
picture = user.picture
@cache.set("/avatars/#{user.avatar}", picture)
end
@cache.quit
cloning / challenging is important as apache / phusion will have connection sharing issues when it forks it, and if we don't close our connections they will keep growing until the file descriptors run out.
I am starting to follow memcache closely to see if I can track my problems, but any suggestions? Should we just get rid of S3 ??
source to share
If I understand this correctly, you store image files in memcached, supported by S3.
Why not just link to images directly from S3, and set the Expires HTTP headers on them so that they are not pulled by the client every time, this would have two benefits:
-
Pages will load faster as browsers will pull in page components from multiple domains.
-
Simplification of the architecture.
source to share
You can use Amazon CloudFront for static resources that will be fetched by the client's browser (e.g. image, static html, css, javascript). Content Delivery Network (CDN) services address latency issues for this type of data; below is a description of the service:
Amazon CloudFront is a web-based content delivery service. It integrates with other Amazon web services to give developers and enterprises an easy way to distribute content to end users with low latency, high data rates, and no commitment. Amazon CloudFront delivers your content using a global network of edge locations. Requests for your objects are automatically routed to the closest edge location, so content is delivered with maximum performance. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3), which reliably stores the original, final versions of your files. Like other Amazon web services, there are no contracts or monthly commitments to use Amazon CloudFront, you only pay for the same or less content.how much you actually get through the service.
Regards, Sirmak
source to share