Statements / Requirements

  1. We need a way for the mobile applications to report sporadic events back to the mothership.
  2. No ‘naked’ end-points
    Need to prevent end users being able to log information directly into ElasticSearch (ES). Elasticsearch Shield may be a solution.
  3. There is a proposed JSON format for incomming events.
    Need to see if this has obvious keys etc or do we let ES create UUID’s for us.
  4. Rate limiting
    Probably a good idea to consider this in the final solution.
  5. Consumable by Kibana
    However, this note does not consider this mechanism further, the act of storing in ES is considered necessary & sufficient.

None requirements

  1. Error logging. We have existing commercial solutions for this. Crashlitics etc
  2. Only consider getting information into ES, not now that data is further queried, migrated etc

Questions

  1. Do we have consider migration of data between solution?
  2. Is it acceptable to ‘lose’ (or even deliberatly discard) data? i.e. Rate Limiting
  3. If we are to rate limit it it must be server side and should be automated.

Areas for investigation

  1. DNS - whatever.example.com Don’t even need to use example.com as service is for internal use only.
  2. Expected data load both mean, mode and max.
    Are we expected to log ‘sitedown’ type events where there may be significant numbers of calls / seconds. [iOS app has > 4,000 concurrent users] There needs to be a method to all-but remove the logging so that times of significant load but not cause unexpected results.

Assumptions

  1. This is a fire-and-forget end point. The application will issues a POST / PUT (As decided) and then ignore any result. Effectivly an async POST. Any actual response will be discared and data lite.
  2. The apps (iOS/Android) will not rate limit themselves, we need to do it for them.

Technologies

  1. ElasticSearch / Kibana
  2. Azure (To host ElasticSearch)
  3. golang for F/E end-point and differentiator

Code

Take a look in $/Event Reporting/... for some HTML & golang sample code for a basic Elastic Search solution.

Google Compute Engine

PaaS - Competitor for AWS etc


Event Logging Service

Data that needs to be recorded when an error or another sort of event is tracked by the app:

  • systemContext
    • channel: native / mobileweb / web
      Should we have different end point for each to load balance for ‘free’?
    • platform: ios / android
    • osVersion: 8.1
    • deviceType: iphone / ipad
    • deviceModel: ipad3,6
    • locale: en-GB / fr-FR
    • dateTime: 2014-07-16T19:20:30
  • appContext
    • name: theapp
    • version: 2.5
    • build: 152
    • store: com
    • country: uk / de / fr
    • language: en-GB
    • sizeSchema: uk
    • currency: gbp
    • screen: productDetails
  • userContext
    • customerId
    • email (privacy?)
    • GUID
    • provider: email / facebook / google / twitter
  • event
    • type: info / error / crash
    • id: api_saveforlater_failed
    • description

Example JSON Document

{
    "systemContext": {
        "channel": "native",
        "platform": "ios",
        "osVersion": "8.1",
        "deviceType": "iphone",
        "deviceModel": "iphone3,6",
        "locale": "en-GB",
        "dateTime": "2014-07-16T19:20:30"
    },
    "appContext": {
        "name": "theapp",
        "version": "2.5",
        "build": "152",
        "store": "com",
        "country": "uk",
        "language": "en-GB",
        "sizeSchema": "uk",
        "currency": "gbp",
        "screen": "productDetails"
    },
    "userContext": {
        "id": "123",
        "email": "email@example.com",
        "guid": "kjfhgsdhjfdskjhfsdkfj",
        "provider": "facebook"
    },
    "event": {
        "type": "error",
        "id": "api_saveforlater_failed",
        "description": "bad request"
    }
}

Managing data on ElasticSearch

Most of the sublties of ES can be ignored, for example coping with conflicts and partial updates.

PUT /{index}/{type}/{id}

index is logical namespace, we’ll use THEapp. type is a data type (in the lossest sence), we’ll use event.

Let ES handle ids.

We could use part of the systemContext (see below) to filter these (By channel or platform). This would allow for load balance - we could even use different end points

native / mobileweb / web & info / error / crash

DNS - something like events.example.com.

Deleting a document

DELETE /THEapp/event/AUsNOr7Qs0fGGvQ_0pBC

Getting all documents

POST _search
{
"query": {
    "match_all": {}
}
}

Querying and filtering

POST /THEapp/event/_search
{
"query": {
    "query_string": {
            "query": "email@example.com",
            "fields": ["userContext.email"]
        }
}
}

POST /THEapp/event/_search
{
    "query": {
        "filtered": {
            "query": {
                "match_all": {
                }
            },
            "filter": {
                "term": { "userContext.id": "123" }
            }
        }
    }
}

Indexing a new document

POST /THEapp/event
{
    "systemContext": {
        "channel": "native",
        "platform": "ios",
        "osVersion": "8.1",
        "deviceType": "iphone",
        "deviceModel": "iphone3,6",
        "locale": "en-GB",
        "dateTime": "2014-07-16T19:20:30"
    },
    "appContext": {
        "name": "THEapp",
        "version": "2.5",
        "build": "152",
        "store": "com",
        "country": "uk",
        "language": "en-GB",
        "sizeSchema": "uk",
        "currency": "gbp",
        "screen": "productDetails"
    },
    "userContext": {
        "id": "123",
        "email": "email@example.com",
        "guid": "kjfhgsdhjfdskjhfsdkfj",
        "provider": "facebook"
    },
    "event": {
        "type": "error",
        "id": "api_addtobag_failed",
        "description": "bad request"
    }
}

TODO

  • Get a simple golang GAE app prototyped - 50% throttled etc.
    App should receive JSON add forward (& forget) request to www.example.com/THEapp/event/.
  • ElasticSearch instance in Azure - nothing flash, using MSDN $$$’s
  • Sample Web App to inject JSON.

Cf.