Google App Engine Datastore

An application creates a new entity of a given kind by calling the constructor of the model class:

from google.appengine.ext import db
class Pet(db.Model):
    name = db.StringProperty(required=True)
    type = db.StringProperty(required=True, choices=set(["cat", "dog", "bird"]))
    birthdate = db.DateProperty()
    weight_in_pounds = db.IntegerProperty()
    spayed_or_neutered = db.BooleanProperty()
    owner = db.UserProperty()
    date_added = db.DateTimeProperty(auto_now_add=True)

pet = Pet(name="Fluffy", type="cat", owner=users.get_current_user())
pet.weight_in_pounds = 24
pet.put()

The new entity is not created in the datastore until it is saved. There are two ways to save entity to the datastore:

pet.put()           // calling the put() method on the instance
db.put(pet)       // passing the instance to the db.put() function

A query returns entities in the form of instances of the model classes that can be modified and put back into the datastore:

user_pets = db.GqlQuery("SELECT * FROM Pet WHERE pet.owner = :1", users.get_current_user())
for pet in user_pets:
    pet.spayed_or_neutered = True
db.put(user_pets)

Note: db.put works on multiple entities, not just a single entity.

A query can:

  1. fetch an entity from the datastore by using its key, or by performing a query that matches the entity's properties.
  2. return zero or more entities
  3. filter results using conditions that entity property values must meet
  4. return the results sorted by property values
  5. limit the scope to entities with a given ancestor
  6. return the results sorted by property values
  7. limit the number of results returned by the datastore

The Model class methods:

.kind()            // Returns the kind of the model, usually the name of the Model subclass.
.properties()  // Returns a dictionary of all of the properties defined for this model class.
.key()             // Returns the datastore Key for this model instance.
.put()             // Stores the model instance in the datastore. If the model instance is newly created and has never been stored, 
                      // this method creates a new data entity in the datastore. Otherwise, it updates the data entity with the current property values.
.delete()         // Deletes the model instance from the datastore. If the instance has never been put(), the delete does nothing.
.is_saved()     // Returns True if the model instance has been put() into the datastore at least once since it was created. 
                       // It does not check if the instance's properties have been updated since the last time it was put()
.dynamic_properties()  // Returns a list of the names of all of the dynamic properties defined for this model instance. 
                                     // This only applies to instances of Expando classes. For non-Expando model instances, this returns an empty list.
.parent()         // Returns a model instance for the parent entity of this instance, or None if this instance does not have a parent.
.parent_key()  // Returns the Key of the parent entity of this instance, or None if this instance does not have a parent.
.to_xml()         // Returns an XML representation of the model instance.  Property values conform to the Atom and GData specifications.
keyname = "some_key"
s = Story.get_or_insert(keyname, title="The Three Little Pigs")

A Model instance does not have a corresponding entity in the datastore until it is put() for the first time, either explicitly so or via Model.get_or_insert().

The Query Interface:

The all() method on a Model (or Expando) class returns a Query object that represents a query for all entities of the corresponding kind. The application prepares the query by calling the filter(), order(), and ancestor() methods on the object:

class Story(db.Model):
  title = db.StringProperty()
  date = db.DateTimeProperty()

query = Story.all()

query.filter('title =', 'Foo')
query.order('-date')
query.ancestor(key)

# These methods can be chained together on one line.
query.filter('title =', 'Foo').order('-date').ancestor(key)

The GqlQuery Interface:

The GqlQuery class constructor takes a GQL query string and optional parameter bindings. The query string specifies the kind, and the filters, sort orders and ancestor conditions. The query string can also include a limit and offset:

# Parameters can be bound with positional arguments.
query = db.GqlQuery("SELECT * FROM Story WHERE title = :1 AND ANCESTOR IS :2 ORDER BY date DESC", 'Foo', key)

# Or, parameters can be bound with keyword arguments.
query = db.GqlQuery("SELECT * FROM Story WHERE title = :title AND ANCESTOR IS :parent ORDER BY date DESC", title='Foo', parent=key)

# String, number and Boolean values can be literal values in the string.
query = db.GqlQuery("SELECT * FROM Story WHERE title = 'Foo' AND ANCESTOR IS :parent ORDER BY date DESC", parent=key)

The gql() method of a Model class prepares a GqlQuery object from a string. The string is the GQL string starting with the SELECT * FROM Model, which is normally omitted because this part is implied:

query = Story.gql("WHERE title = :title AND ANCESTOR IS :parent ORDER BY date DESC", title='Foo', parent=key)

Query and GqlQuery objects do not execute the query until the application tries to access the results. When the application try to accesses the results, the query is executed, and results are loaded into memory as instances of the model class for the query. Both query classes provide two ways to execute the query and access results: the fetch() method, and the iterator interface.

The fetch() method takes a maximum number of results to fetch (the limit), and an optional number of results to skip (the offset). The method executes the query, then fetches results until it has fetched the limit or there are no more results. Once the results are loaded into memory, it skips to the offset if one was specified, then returns the requested results as a list of model instances. The full query is executed for each call to fetch(). The offset does not affect the number of results fetched from the datastore. All results up to the limit are fetched and stored in memory. The offset only affects what is returned by the fetch() method.

results = query.fetch(10)

The limit and offset given to the fetch() method override any limit and offset specified in a GQL query string.

If the query object is used as an iterator, the query is executed with no limit or offset, the results are loaded into memory, and the value returned is an iterator over the results. The iterator yields instances of the model class.

for result in query:

The Query and GqlQuery classes also have these interesting methods:

.get()  // Executes the query, then returns the first result, or None if the query returned no results.  
           // Implies a "limit" of 1, and overrides the LIMIT clause of the GQL query, if any. At most 1 result is fetched from the datastore.
.count(limit)  // Returns the number of results this query fetches.  count() is somewhat faster than retrieving all of the data 
                    // by a constant factor, but the running time still grows with the size of the result set. It's best to only use count() 
                    // in cases where the count is expected to be small, or specify a limit.  count() returns a maximum of 1000.

db Functions:

db.get(keys)     // Gets the entity or entities for the given key or keys, of any Model.
                         // keys: A Key object or a list of Key objects.  If one Key is provided, the return value is an instance of the 
                         // appropriate Model class, or None if no entity exists with the given Key.  If a list of Keys is provided, the 
                         // return value is a corresponding list of model instances, with None values when no entity exists for a corresponding Key.
db.put(models) // Puts one or more instances into the datastore.
                         // models: A model instance or a list of model instances to store.  If multiple model instances are given, 
                         // they may be in more than one entity group. Within each entity group, all entities belonging to that group will 
                         // be written in a single transaction.  An exception will always be raised if any error occurs during the operation, 
                         // even if some of the entities actually were written. This may happen if the entities in the call span multiple entity
                         // groups. If the call returns without raising an exception, then all of the entities were written successfully.
                         // Returns the Key object (if one model instance is given) or a list of Key objects (if a list of instances is given) 
                         // that correspond with the stored model instances.
db.delete(models) // Deletes one or more model instances from the datastore.
                              // models: A model instance, a Key for an entity, or a list of model instances or keys of entities to delete.  
                              // If multiple keys are given, they may be in more than one entity group. Within each entity group, all 
                              // entities belonging to that group will be deleted in a single transaction.  An exception will always be 
                              // raised if any error occurs during the operation, even if some of the entities actually were deleted. 
                              // This may happen if the keys in the call span multiple entity groups. If the call returns without raising 
                              // an exception, then all of the entities were deleted successfully.
db.run_in_transaction(function, *args, **kwargs) // Runs a function containing datastore updates in a single transaction. 
                          // If any code raises an exception during the transaction, all datastore updates made in the transaction are rolled back....

Getting an Entity Using a Key:

After an entity is stored in the datastore, the entity has a unique key. The put() method of an instance and the db.put() function return the Key of the stored entity. After a model instance has been stored for the first time, the key() method of the instance returns the Key for the instance.

entity.put()
key = entity.key()

entity = db.get(key)

The db.get() function fetches an entity from the datastore for a Key (or list of Keys).

Keys can be encoded as strings for passing around outside of the application. To convert a string-encoded key back to a Key object, the application passes the string to the Key constructor:

obj = MyModel(name="Foo")
self.response.write('<a href="/view?key=%s">%s</a>' % (str(obj.key()), obj.name()))

key_name = self.request.get('key')
obj = db.get(db.Key(key_name))

The string encoding of a Key is opaque, but not encrypted. If your application needs keys to not be guessable, you should further encrypt the string-encoded Key before sending it to the user.

A key can be encoded to a string by passing the Key object to str() (or calling the object's str() method). A string-encoded key is an opaque value using characters safe for including in URLs. The string-encoded key can be converted back to a Key object by passing it to the Key constructor (the encoded argument).

A string-encoded key can be converted back to the raw key data. This makes it easy to guess other keys when one is known. While string-encoded key values are safe to include in URLs, an application should only do so if key guessability is not an issue.

Deleting an Entity:

An application can delete an entity from the datastore using the delete() method of the instance or a Key. The delete() method of the instance deletes the corresponding entity from the datastore. The db.delete() function takes a Key or list of Keys and deletes the entity (or entities) from the datastore:

q = db.GqlQuery("SELECT * FROM Message WHERE create_date < :1", earliest_date)
results = q.fetch(10)
for result in results:
  result.delete()

# or...

q = db.GqlQuery("SELECT * FROM Message WHERE create_date < :1", earliest_date)
results = q.fetch(10)
db.delete(results)

Deleting an entity does not change any Key values in the datastore that may have referred to the entity. If your application may attempt to de-reference a Key value for a deleted entity, the application should do so using db.get(), then test the return value before accessing properties.

Using model, a property can be configured, such as if the property is required, or a default value to use if none is provided. See above.

Because validation occurs when the instance is constructed, any property that is configured to be required must be initialized in the constructor.

An expando model can have both fixed and dynamic properties. The model class simply sets class attributes with Property configuration objects for the fixed properties. The application creates dynamic properties when it assigns them values.

Dynamic properties are not defined in the model class.

The datastore supports two value types for storing text: short text strings (shorter than 500 bytes), and long text strings (longer than 500 bytes). Short strings are indexed and can be used in query filter conditions and sort orders. Long strings are not indexed and cannot be used in filter conditions or sort orders.

A short string value can be either a unicode value or a str value. If the value is a str, an encoding of 'ascii' is assumed. To specify a different encoding for a str value, you can convert it to a unicode value with the unicode() type constructor, which takes the str and the name of the encoding as arguments. Short strings can be modelled using the StringProperty class:

class MyModel(db.Model):
  string = db.StringProperty()

obj = MyModel()

# Python Unicode literal syntax fully describes characters in a text string.
obj.string = u"kittens"

# unicode() converts a byte string to a Unicode value using the named codec.
obj.string = unicode("kittens", "latin-1")

# A byte string is assumed to be text encoded as ASCII (the 'ascii' codec).
obj.string = "kittens"

# Short string properties can be used in query filters.
results = db.GqlQuery("SELECT * FROM MyModel WHERE string = :1", u"kittens")

Long strings can be modelled using the TextProperty class. Its constructor takes either a unicode value, or a str value and optionally the name of the encoding:

class MyModel(db.Model):
  text = db.TextProperty()

obj = MyModel()

# Text() can take a Unicode value.
obj.text = db.Text(u"lots of kittens")

# Text() can take a byte string and the name of an encoding.
obj.text = db.Text("lots of kittens", "latin-1")

# If no encoding is specified, a byte string is assumed to be ASCII text.
obj.text = db.Text("lots of kittens")

# Text properties can store large values.
obj.text = db.Text(open("a_tale_of_two_cities.txt").read(), "utf-8")

BlobProperty:

The datastore also supports "blobs" (string of bytes) As with long text strings, blobs are not indexed, and cannot be used in query filters or sort orders. A Blob instance takes a str value as an argument to its constructor. Blobs are modelled using the BlobProperty class:

class MyModel(db.Model):
  blob = db.BlobProperty()

obj = MyModel()
obj.blob = db.Blob(open("image.png").read())

ListProperty:

ListProperty can contain values of any of the value types supported by the datastore. A single list property may even have values of different types. Order is preserved. When entities are returned by queries, list properties will have values in the same order as when they were stored. The ListProperty class models a list, and enforces that all values in the list are of a given type:

class MyModel(db.Model):
  numbers = db.ListProperty(long)

obj = MyModel()
obj.numbers = [2, 4, 6, 8, 10]

A query filter on a list property tests the given value against members of the list. The condition is true if at least one member of the list meets the condition:

# Get all entities where numbers contains a 6.
results = db.GqlQuery("SELECT * FROM MyModel WHERE numbers = 6")

Query filters only operate on the list members. There is no way to test two lists for similarity in a query filter.

For storing non-text byte strings, use db.Blob values. The bytes of a blob string are preserved when they are stored and retrieved. You can declare a property that is a list of blobs as ListProperty(db.Blob).

ReferenceProperty:

A property value can contain the key of another entity. The value is a Key instance. The ReferenceProperty enforces that all values refer to entities of a same kind. Assigning a model instance to a ReferenceProperty property automatically uses its key as the value:

class FirstModel(db.Model):
  prop = db.IntegerProperty()

class SecondModel(db.Model):
  reference = db.ReferenceProperty(FirstModel)

obj1 = FirstModel()
obj1.prop = 42
obj1.put()

obj2 = SecondModel()

# A reference value is the key of another entity.
obj2.reference = obj1.key()

# Assigning a model instance to a property uses the entity's key as the value.
obj2.reference = obj1
obj2.put()

A ReferenceProperty property value can be used as if it was the referenced entity. If the referenced entity is not in memory, using the property as an instance automatically fetches the entity from the datastore:

obj2.reference.prop = 999
obj2.reference.put()

results = db.GqlQuery("SELECT * FROM SecondModel")
another_obj = results.fetch(1)[0]
v = another_obj.reference.prop

When an entity whose key is the value of a reference property is deleted, the reference property does not change. A reference property value can be a key that is no longer valid. If an application expects that a reference could be invalid, it can test for the existence of the object using an if statement:

obj1 = obj2.reference
if not obj1:

Back-references. When a model has a ReferenceProperty to another model, each referenced entity gets a property whose value is a Query that returns all of the entities of the first model that refer to it. The name of the back-reference property defaults to modelname_set (with the name of the model class in lowercase letters, and "_set" added to the end), and can be adjusted using the collection_name argument to the ReferenceProperty constructor.

Automatic referencing and dereferencing of model instances, type checking and back-references are only available using the ReferenceProperty. Keys stored as values of Expando dynamic properties or ListProperty values do not have these features.

Property Names:

The datastore reserves all property names begining and ending with two underscore characters (*). Attributes of model instances whose names begin with an underscore (_) are ignored, and are not saved to the datastore entity. This allows you to store values on the model instance for temporary internal use without affecting the data saved with the entity.

http://code.google.com/appengine/docs/datastore/typesandpropertyclasses.html
The Model Class - A model is a Python class that inherits from the Model class.
The Property Class - Each class attribute is an instance of a subclass of the Property class.
Expando Model - Sometimes it is useful for an entity to have properties that aren't necessarily like the properties of other entities of the same kind. (Say one entity has this particular property as integer, another entity of the same kind (model class) has the same property as string).
Key class
Transaction
http://code.google.com/appengine/docs/datastore/entitiesandmodels.html

Kind is the model class. Entity (a row in relational database table) is an instance of the model class.

Can you directly delete or update entities from the datastore without first loading it using a SELECT?

What is a parent entity? How do I use it?

When should I manually tune indexes?

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License