Glue

• Serverless ETL (Spark-based or Python shell)
• Centralized Data Catalog for metadata
• Automatic schema discovery via crawlers
• Visual ETL authoring with Glue Studio
• Built-in transforms and libraries to speed common ETL tasks

​

Key components and behaviors

• Crawlers perform automatic type inference. If the inferred schema is incorrect, refine the schema in the Data Catalog or add custom classifiers.
• Use job bookmarks to enable incremental ETL and avoid reprocessing already-processed data.
• Glue integrates tightly with the AWS analytics ecosystem (Athena, EMR, Redshift Spectrum, QuickSight).

​

Typical Glue workflow

1. Point a crawler at one or more data sources (S3, JDBC, DynamoDB, etc.).
2. The crawler discovers the schema and creates/updates tables in the Glue Data Catalog.
3. Author a Glue job (PySpark or Scala for Spark jobs, or Python shell for light tasks) or use the Glue Studio visual editor.
4. Trigger the job on-demand, on a schedule, or via events (EventBridge / CloudWatch Events).
5. The job extracts data, applies transforms, and writes results to the target (S3 data lake, Redshift, etc.).
6. Use the Glue Data Catalog as the metadata source for Athena, EMR, Redshift Spectrum, or visualization tools like QuickSight.

​

Examples

aws glue start-crawler --name my-crawler

# python
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
import sys

# Parse job arguments when running in Glue
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read from the Glue Data Catalog
datasource = glueContext.create_dynamic_frame.from_catalog(
    database="my_database",
    table_name="my_table"
)

# Apply a transform (example: filter rows)
from awsglue.transforms import Filter
filtered = Filter.apply(frame=datasource, f=lambda row: row["status"] == "ACTIVE")

# Write to S3 in Parquet format
glueContext.write_dynamic_frame.from_options(
    frame=filtered,
    connection_type="s3",
    connection_options={"path": "s3://my-target-bucket/processed/"},
    format="parquet"
)

job.commit()

​

Integrations and outputs

• Common targets: Amazon S3 and Amazon Redshift.
• The Glue Data Catalog is consumable by: Amazon Athena, Amazon EMR, Amazon Redshift Spectrum.
• Downstream visualization and BI tools (e.g., Amazon QuickSight) can use datasets produced by Glue jobs.

​

Transforms and libraries

• Field operations: rename, drop, add
• Filtering, mapping, joining, aggregations
• Format conversions (CSV, JSON, Parquet, ORC)
• Schema reconciliation when source schemas vary

​

Best practices and tips

• Use the Glue Data Catalog as the single source of truth for table metadata across analytics services.
• Do not rely solely on automated schema inference for production-critical pipelines — validate inferred schemas or supply explicit classifiers.
• Use job bookmarks for incremental processing to prevent duplicate processing.
• Start with Glue Studio for rapid prototyping; export and refine job code when you need advanced custom logic or performance tuning.
• Monitor job runs and set up alerts using CloudWatch and EventBridge for production reliability.

​

Links and references

• AWS Glue product page
• AWS Glue documentation
• Amazon S3
• Amazon Redshift
• Amazon Athena
• Amazon EMR
• Amazon QuickSight