untrusted app XXX cannot access app helloworld's data

BadRequestError: untrusted app XXX cannot access app helloworld's data


Reference Property の値( =Key ) をbulkupload
・・・
('stock', datastore_types.Key),
・・・
しようとして発生。

仮に upload できたとしても、意味がないのでコメントアウト。

max execution time

put にしろ fetch にしろ１回の処理に制限がある。

bulkload は 100 件
fetch は 1,000 件
delete も 100 件程度


> Is there any limit of execution time of scripts? If there is, how long
> is this max execution time for a function?

A few seconds, eight or ten seconds at the most.

> Is there any way to set the max_execution_time or functions like
> set_time_limit() in php?
I don't think so.

> If I want my script to execut in a long time to finish its job, maybe
> 20 mininuts, just looks like a service, is there any way in App
> Engine ?

Nope. Google App Engine is only designed for applications that can
respond to requests in under 300 ms or so.


http://groups.google.com/group/google-appengine/browse_thread/thread/08548c43d692af63?hl=en

fetch() returns a maximum of 1000 results

SDK 環境ではこの
1000件以上のデータがあっても、1000件までしか、検索結果を得ることができない
などという制限はなかったので、
実際にようやく bulkloader でデータを 投入してみてはじめて気がついた。
SELECT * FROM model limit 100 offset 1000
のようなことをすれば、1000 の先のデータにアクセスできるのか？　とも思ったが、そんなことはできなかった。

http://code.google.com/appengine/docs/datastore/gqlqueryclass.html

• The query has performance characteristics that correspond linearly with the offset amount plus the limit.

Note: fetch() returns a maximum of 1000 results. If more than 1000 entities match the query, and either no limit is specified or a limit larger than 1000 is used, only the first 1000 results are returned by fetch().

Google I/O session videos posted with slides の
Working with Google App Engine Models に

　Store counts を推奨するようなことが書かれて
いたので、ちょっと気になってはいたがけれども。
1000 件以内に収まる条件で管理しないと
どこまでデータがあるのかさえわからなくなるとは。

Importing UTF-8 Data with Bulkloader

Importing UTF-8 Data
http://groups.google.com/group/google-appengine/browse_thread/thread/d4cf3013483220b5
を参考にいろいろ試したが、以下のパターンで UTF-8 が登録できたが、
str とタイプが決まってしまて、　datastore_types.Text　にできなかったので
500 バイト以上の UTF-8 データについては別途検討。

myloader.py

#from google.appengine.ext import bulkload
import bulkload
・・・
class PersonLoader(bulkload.Loader):
 def __init__(self):
   # Our 'Person' entity contains a name string and an email
   bulkload.Loader.__init__(self, 'Person',
             [
#  　　　　   ('name', str),
 　　　　　  ('name', lambda x: unicode(x,'utf-8')),
             ('email', datastore_types.Email),
             ])

bulkload をフォルダごとProjectフォルダにコピーして、__init__.py を加工。
bulkload/__init__.py

・・・
   buffer = StringIO.StringIO(data.encode('utf-8'))
・・・

('name', lambda x: unicode(x,'utf-8'))　をやめて、以下を追加すると
日本語がきれいに抜けて、英数字だけになる。

#     val =  unicode(val, errors='ignore')
     entity[name] = converter(val)

bulkload/__init__.py

c:/Program Files/Google/google_appengine/google/appengine/ext/bulkload/
__init__.py

"""A mix-in handler for bulk loading data into an application.

For complete documentation, see the Tools and Libraries section of the
documentation.

To use this in your app, first write a script, e.g. bulkload.py, that
instantiates a Loader for each entity kind you want to import and call
bulkload.main(instance). For example:

person = bulkload.Loader(
  'Person',
  [('name', str),
   ('email', datastore_types.Email),
   ('birthdate', lambda x: datetime.datetime.fromtimestamp(float(x))),
  ])

if __name__ == '__main__':
  bulkload.main(person)

See the Loader class for more information. Then, add a handler for it in your
app.yaml, e.g.:

  urlmap:
  - regex: /load
    handler:
      type: 1
      path: bulkload.py
      requires_login: true
      admin_only: true

Finally, deploy your app and run bulkload_client.py. For example, to load the
file people.csv into a dev_appserver running on your local machine:

./bulkload_client.py --filename people.csv --kind Person --cookie ... \
                     --url http://localhost:8080/load

The kind parameter is used to look up the Loader instance that will be used.
The bulkload handler should usually be admin_only, so that non-admins can't use
the shell to modify your app's data. The bulkload client uses the cookie
parameter to piggyback its HTTP requests on your login session. A GET request
to the URL specified for your bulkload script will give you a cookie parameter
you can use (/load in the example above).  If your bulkload handler is not
admin_only, you may omit the cookie parameter.

If you want to do extra processing before the entities are stored, you can
subclass Loader and override HandleEntity. HandleEntity is called once with
each entity that is imported from the CSV data. You can return one or more
entities from HandleEntity to be stored in its place, or None if nothing
should be stored.

For example, this loads calendar events and stores them as
datastore_entities.Event entities. It also populates their author field with a
reference to the corresponding datastore_entites.Contact entity. If no Contact
entity exists yet for the given author, it creates one and stores it first.

class EventLoader(bulkload.Loader):
  def __init__(self):
    EventLoader.__init__(self, 'Event',
                         [('title', str),
                          ('creator', str),
                          ('where', str),
                          ('startTime', lambda x:
                            datetime.datetime.fromtimestamp(float(x))),
                          ])

  def HandleEntity(self, entity):
    event = datastore_entities.Event(entity.title)
    event.update(entity)

    creator = event['creator']
    if creator:
      contact = datastore.Query('Contact', {'title': creator}).Get(1)
      if not contact:
        contact = [datastore_entities.Contact(creator)]
        datastore.Put(contact[0])
      event['author'] = contact[0].key()

    return event

if __name__ == '__main__':
  bulkload.main(EventLoader())
"""

---
    bulkload.Loader.__init__(self, 'Greeting',
           [
                          ('author',   datastore_types.users.User ),
     ('content', str ), 
     ('curl1',   str ), 
     ('cmapinfo', datastore_types.Text), 
     ('date',    lambda x: datetime.datetime.strptime(x,'%Y-%m-%d %H:%M:%S')),
     ('dateJST', lambda x: datetime.datetime.strptime(x,'%Y-%m-%d %H:%M:%S')),
           ])

--

C:\google\bulkload>c:\google\bulkload\bulkload_client.py --filename greeting_test.csv --kind Greeting --url http://xxxx.appspot.com/load02

model.properties()

kind =(Greeting(),Blog() )

for k in kind:

  print "DROP TABLE IF EXISTS `" + k.kind() +"`;"
  print "Create table `" + k.kind() + "` ("
  print "    `key`        char(36) not null,"
  print "    `id`        int,"
  print "    `key_name`    char(12),"
  p =  k.properties()
  for pp in p:
    attr = str(p[pp]).split('google.appengine.ext.db.')[1]
    attr = attr.split(' object')[0]
    sql_attr = attr
    if attr == "IntegerProperty"    : sql_attr = "int"
    if attr == "StringProperty"     : sql_attr = "varchar(500)"
    if attr == "ReferenceProperty"    : sql_attr = "char(36)"
    if attr == "DateTimeProperty"    : sql_attr = "datetime"
    if attr == "UserProperty"        : sql_attr = "varchar(128)"
    if attr == "TextProperty"        : sql_attr = "text"
    print "`" + pp + "`" + "    " + sql_attr + ",    #" + attr
  print ") #  remove last `,` !!!"

Local or Cloud

ようやく Cloud に 「Create and Application」できた。
しかし app.yaml の application: の部分を変更して、起動したことで
Local の datastore が初期化されてしまった。
バックアップがあるので、２，３日前の状態には戻せるが...

#!-*- coding:utf-8 -*-
import cgi
import wsgiref.handlers
import os

from google.appengine.ext import webapp
class MainPage(webapp.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/html'
self.response.out.write(u'Hello, webapp World! こんにちわ<br />' )
self.response.out.write(os.environ["PATH_INFO"]   + "<br />")
self.response.out.write(os.environ["TZ"]          + "<br />" )
self.response.out.write(os.environ["SERVER_NAME"] + "<br />")

-----
Hello, webapp World! こんにちわ
/hw0
UTC
localhost