Skip to main content

Projects & Git

Project is the primary unit of development and deployment to production in Prophecy.

A project contains

  • Data pipelines that read, transform and write data using Spark
  • Datasets point to the data that is read and written to by the Data Pipelines
  • Jobs run Data Pipelines based on a schedule

Project is Code on Git

A project is code on git. This means that within a project, the business logic of all the assets including data pipelines, datasets, and jobs is stored as code on Git. This might be a repository on Github or a folder in a repository.

Project is code

Development & Deployment

Prophecy provides a standard and recommended mechanism for using Git based development (though other mechanisms are possible - including fork based development in our Enterprise product)

Project deply

Here are the steps explained:

1. Create new project

First you'll create a new project as shown below. It is strongly recommended that you connect to your Git repository to ensure that there is a secure copy of the code that you have direct access to.

New project

2. Create, edit & commit the pipeline

When you create a new pipeline, you have to choose the branch where it will be created - an existing one or a new one.

Then you will develop this pipeline - you will make changes & commit them in this branch multiple times. The commit dialog opens when you click the bottom bar - orange color indicates uncommitted changes.


Now you've made all the changes back and are ready to push them back to the main repository

3. Merge your changes

When you create a new workflow, a default branch is created for you.