Memcache API

http://code.google.com/appengine/docs/memcache/

日本語訳　http://d.hatena.ne.jp/technohippy/20080717#1216393318



memcache.add( key, value, time=xx, min_compress_len=0)

time　設定した有効時間(単位秒)以後は再度、検索処理を行なう

Optional expiration time, either relative number of seconds from current time (up to 1 month), or an absolute Unix epoch time. By default, items never expire, though items may be evicted due to memory pressure. Float values will be rounded up to the nearest whole second. 　　

• set()
• set_multi()
• get()
• get_multi()
• delete()
• delete_multi()
• add()
• replace()
• incr()
• decr()
• flush_all()
• get_stats()
  ex.
  {'hits': 34, 'items': 5, 'bytes': 45587, 'oldest_item_age': 1800, 'misses': 21, 'byte_hits': 946479}
  

Debug デバッグ

quick and dirty: きれいではなけれども簡単な方法(開発環境用)
* Cloud ではログにエラー出力されるの危険

import sys
print >>sys.stderr, "xxxxxxx"

正しくは

import logging
logging.debug("xxxxxxx")

def main():
# Set the logging level in the main function
# See the section on Requests and App Caching for information on how
# App Engine reuses your request handlers when you specify a main function
logging.getLogger().setLevel(logging.DEBUG)
application = webapp.WSGIApplication([('/', MainPage),
                                      ('/sign', Guestbook)],
                                     debug=True)
webapp.util.run_wsgi_app(application)

if __name__ = '__main__':
main()

参考
http://code.google.com/appengine/articles/logging.html
・http://code.google.com/appengine/docs/python/logging.html
・http://groups.google.com/group/google-appengine/browse_thread/thread/a67752ac402bb21e/345e203a5bdd0750?lnk=gst&q=debug+#345e203a5bdd0750
・Django Middleware で Traceback をコンソールに出力する
http://yamashita.dyndns.org/blog/django-middleware-traceback/

UnicodeEncodeError と webapp.RequestHandler

class MainPage(webapp.RequestHandler):
　の中で
str.decode("utf-8",'ignore')
　とすると
UnicodeEncodeError
　となるようなのですが、
webapp.RequestHandler　を介さない別のところで処理している場合、
エラーは発生していない。

UnicodeEncodeError　はまだよく理解できていないところがありますが。

メモ

UnicodeEncodeError: 'ascii' codec can't encode characters in position 422-424
: ordinal not in range(128)

Unicode 文字列をバイト列に符号化 (encode) するとき， range(128) つまり 0 から 127 までの文字コードしか扱えない 'ascii' エンコーディングが使われ，日本語文字に対して例外 UnicodeEncodeError が起こったもの
http://www.okisoft.co.jp/esc/cygwin-15a.html

webapp を調べたよ (前編)　- Google App Engine
http://d.hatena.ne.jp/hamatsu1974/20080422/1208802967

DeadlineExceededError

DeadlineExceededError のエラー対応を検討

対応例
http://stage.vambenepe.com/archives/category/implementation

結果　google/appengine/runtime/apiproxy.py　を参考に

from google.appengine import runtime
from google.appengine.runtime import apiproxy_errors
from google3.apphosting.runtime import _apphosting_runtime___python__apiproxy

これらを import して対応

except runtime.DeadlineExceededError:

File "/base/python_lib/versions/1/google/appengine/runtime/apiproxy.py", line 161, in Wait
  rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
DeadlineExceededError

だけでなく CancelleError というものもある。

File "/base/python_lib/versions/1/google/appengine/runtime/apiproxy.py", line 189, in CheckSuccess
  raise self.exception
CancelledError: The API call datastore_v3.Count() was explicitly cancelled.

Key Limitations: 500 bytes

key_name を設定すると、その分 key が長くなる。
parent を設定すると、親の key が自分の key の先頭に付く。
parent が parent を持っていると、孫の key は長くなる。
限界が 500 byte

従って、当然、 親を設定し、さらにこれに親が設定されていれば
ancestor is 　により祖先を検索条件に指定することができる。
b = db.GqlQuery("select * from Blog where ancestor is :1 ", granpa.key())


Tools for storing data: Keys

* Key corresponds to the Bigtable row for an Entity
* Bigtable accessible as a distributed hashtable
* Get() by Key: Very fast! No scanning, just copying data


* Limitations:
o Only one ID or key_name per Entity
o Cannot change ID or key_name later
o 500 bytes

Building Scalable Web Applications/Building a Blog

from google.appengine.ext import db

class BlogIndex(db.Model):
 max_index = db.IntegerProperty(required=True,default=0)

class BlogEntry(db.Model):
 index = db.IntegerProperty(required=True)
 title = db.StringProperty(required=True)
 body  = db.TextProperty(required=True)


def post_entry(blogname, title, body):
 def txn():
   blog_index = BlogIndex.get_by_key_name(blogname)
   if blog_index is None:
     blog_index = BlogIndex(key_name=blogname)
   new_index = blog_index.max_index
   blog_index.max_index += 1
   blog_index.put()
   new_entry = BlogEntry(key_name=blogname + str(new_index),parent=blog_index, index=new_index,title=title, body=body)
   new_entry.put()
 db.run_in_transaction(txn)

def get_entry(blogname, index):
 entry = BlogEntry.get_by_key_name(blogname + str(index),parent=Key.from_path('BlogIndex',blogname))
 return entry


def get_entries(start_index):
 extra = None
 if start_index is None:
   entries = BlogEntry.gql(
       'ORDER BY index DESC').fetch(POSTS_PER_PAGE + 1)
 else:
   start_index = int(start_index)
   entries = BlogEntry.gql(
       'WHERE index <= :1 ORDER BY index DESC',         start_index).fetch(POSTS_PER_PAGE + 1)   if len(entries) > POSTS_PER_PAGE:
   extra = entries[-1]
   entries = entries[:POSTS_PER_PAGE]
 return entries, extra

Building Scalable Web Applications から

Disk Seek　を意識している。
Oracle などで　Create sequence するように Counter　専用のModel を作成して、
increment("xxxx")するようにしたほうがよいのか。


Writes are expensive!

• Datastore is transactional: writes require disk access
  • Disk access means disk seeks

• Rule of thumb: 10ms for a disk seek
• Simple math:
  • 1s / 10ms = 100 seeks/sec maximum

• Depends on:
  • The size and shape of your data
  • Doing work in batches (batch puts and gets)

Reads are cheap!

• Reads do not need to be transactional, just consistent

• Data is read from disk once, then it's easily cached
• All subsequent reads come straight from memory

• Rule of thumb: 250usec for 1MB of data from memory
• Simple math:
  • 1s / 250usec = 4GB/sec maximum
  • For a 1MB entity, that's 4000 fetches/sec

Tools for storing data: Entities

• Fundamental storage type in App Engine
• Set of property name/value pairs
• Most properties indexed and efficient to query
• Other large properties not indexed (Blobs, Text)

• Think of it as an object store, not relational
  • Kinds are like classes
  • Entities are like object instances
• Relationship between Entities using Keys
  • Reference properties
  • One to many, many to many

Tools for storing data: Entity groups 2

Hierarchical

• Each Entity may have a parent
• A "root" node defines an Entity group
• Hierarchy of child Entities can go many levels deep
  • Watch out! Serialized writes for all children of the root

Datastore scales wide

• Each Entity group has serialized writes
• No limit to the number of Entity groups to use in parallel
• Think of it as many independent hierarchies of data

class CounterConfig(db.Model):
　name       = db.StringProperty(required=True)
　num_shards = db.IntegerProperty(required=True,default=1)

class Counter(db.Model):
　name       = db.StringProperty(required=True)
　count      = db.IntegerProperty(required=True,default=0)

def get_count(name):
　total = 0
　for counter in Counter.gql('WHERE name = :1', name):
　　total += counter.count
　return total

def increment(name):
　config = CounterConfig.get_or_insert(name,name=name)
　def txn():
　　index = random.randint(0, config.num_shards - 1)
　　shard_name = name + str(index)
　　counter = Counter.get_by_key_name(shard_name)
　　if counter is None:
　　　counter = Counter(key_name=shard_name, name=name)
　　counter.count += 1
　　counter.put()
　db.run_in_transaction(txn)

increment("test")
print get_count("test")


def get_count(name):
　total = memcache.get(name)
　if total is None:
　total = 0
　for counter in Counter.gql('WHERE name = :1', name):
　　total += counter.count
　　memcache.add(name, str(total), 60)
　return total

def increment(name):
　config = CounterConfig.get_or_insert(name,name=name)
　def txn():
　　index = random.randint(0, config.num_shards - 1)
　　shard_name = name + str(index)
　　counter = Counter.get_by_key_name(shard_name)
　　if counter is None:
 　　　counter = Counter(key_name=shard_name,name=name)
　　counter.count += 1
　　counter.put()
　db.run_in_transaction(txn)
　memcache.incr(name)

http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine