magpie

Subpackages

• magpie.fetchers
  • magpie.fetchers.generic
  • magpie.fetchers.hackernews
  • magpie.fetchers.github
• magpie.fetch
  • magpie.fetch.retriever
  • magpie.fetch.cache
• magpie.data
  • magpie.data.testdata
• magpie.mock
  • magpie.mock.network
• magpie.task
  • magpie.task.worker
  • magpie.task.monitor
  • magpie.task.queue
  • magpie.task.flower

Submodules

• magpie.network
• magpie.util
• magpie.datamodel_patch
• magpie.logging
• magpie.datamodel
• magpie.config

Package Contents

Functions

init	Initialize the Magpie core processes/threads.
shutdown	Shutdown all the worker processes from the task queue system.

API

magpie.init()

Initialize the Magpie core processes/threads.

In particular, this will:

• start a monitoring thread to get back results asynchronously once tasks have finished computing

• start a celery worker process to consume and process tasks from the queue (if configured)

• start a celery flower task monitoring system (if configured)

magpie.shutdown()

Shutdown all the worker processes from the task queue system.