Welcome to Comet data pipeline’s documentation!

Getting started with Comet Data Pipeline:

User Guide

• Introduction
  • How it works
    • On Premise Data Pipeline
    • Azure Databricks Data Pipeline
    • Data Pipeline on Google Cloud Storage
    • Data Pipeline on BigQuery
• Example
• Configuration
  • Environment variables
  • Airflow DAGs
    • Import DAG
    • Watch DAG
    • Ingestion DAG
• Quick Start
  • Build it
  • Run it

Howto

• Extract
  • The YAML File
  • The Mustache Template
  • The comet extract command
• Load
  • Load Rules
    • Type Rules
    • Domain Rules
    • Schema Rules
    • Write Strategy
  • Load Workflow
    • Import Step
    • Watch Step
    • Ingestion Step
    • Export Step
• Transform

Concepts

• Extract
• Load
  • Domain
  • Schema
  • Metadata
  • Attribute
  • Sink
    • BigQuerySink
    • EsSink
    • JdbcSink
  • RowLevelSecurity
• Transform
  • Job
  • Task
  • Partitioning
  • Clustering
  • Views

Examples

• Extract
• Load
  • Load to Parquet
  • Load to BigQuery
  • Load to SQL Database
  • Load to Elasticsearch
    • Custom ES Template
• Transform
  • Parquet to Parquet
  • Transform Parquet to DSV
  • Transform Parquet to BigQuery
  • BigQuery to BigQuery
  • BigQuery to CSV
  • BigQuery to Parquet
  • Parquet to Elasticsearch
  • BigQuery to Elasticsearch
  • BigQuery to SQL Database
  • Parquet to SQL Database
  • SQL Database to SQL Database

Cloud Specific

• Microsoft Azure
  • Storage Accounts
• Google Cloud Platform
  • Running Locally
• Amazon Web Services

CLI

• bqload
  • Synopsis
  • Description
  • Options
• esload | index
  • Synopsis
  • Description
  • Options
• extract
  • Synopsis
  • Description
  • Options
• import
  • Synopsis
  • Description
  • Options
• infer-schema
  • Synopsis
  • Description
  • Options
• load | ingest
  • Synopsis
  • Description
  • Options
• metrics
  • Synopsis
  • Description
  • Options
• parquet2csv
  • Synopsis
  • Description
  • Options
• cnxload
  • Synopsis
  • Description
  • Options
• transform | job
  • Synopsis
  • Description
  • Options
• watch
  • Synopsis
  • Description
  • Options
• xls2yml
  • Synopsis
  • Description
  • Options

Developer Guide

• How to contribute to Comet Data Pipeline
  • How to contribute to this documentation
  • Contribution for Ebiznext consultants
  • Contributing Issue Reports
  • Contributing to architecture documentation
  • Contributing by Reviewing Changes
• The Review Process
  • Contributing Documentation Changes
  • Contributing Code Changes
• Git workflow
• Merge Request
  • Code Style Guide