0.10.1 (2024-02-05)

Features

• Add support of Incremental Strategies for Kafka connection:

  reader = DBReader(
      connection=Kafka(...),
      source="topic_name",
      hwm=DBReader.AutoDetectHWM(name="some_hwm_name", expression="offset"),
  )
  
  with IncrementalStrategy():
      df = reader.run()
  

This lets you resume reading data from a Kafka topic starting at the last committed offset from your previous run. (#202) - Add has_data, raise_if_no_data methods to DBReader class. (#203) - Updare VMware Greenplum connector from 2.1.4 to 2.3.0. This implies: : * Greenplum 7.x support * Kubernetes support * New read option gpdb.matchDistributionPolicy which allows to match each Spark executor with specific Greenplum segment, avoiding redundant data transfer between Greenplum segments * Allows overriding Greenplum optimizer parameters in read/write operations (#208) - Greenplum.get_packages() method now accepts optional arg package_version which allows to override version of Greenplum connector package. (#208)