Merge PR #11 from beta branch to main#24

Draft

StrongestNumber9 wants to merge 148 commits into

mainfrom

Commits on Jan 29, 2024

Initial setup for Kafka consumer groups.
Tiihott
committed
Implementing idempotent consumer pattern to the Kafka consumer groups (WIP).
Tiihott
committed
Implementing idempotent consumer pattern to the Kafka consumer groups (WIP2).
Tiihott
committed
Implementing Kafka consumed record processing using AVRO serialization and HDFS storage (WIP). Skipping implementing idempotent consumer pattern to the Kafka consumer groups.
Tiihott
committed
Implementing AVRO serialization to Kafka record processing.
Tiihott
committed
Implementing AVRO serialization to Kafka record processing, WIP2.
Tiihott
committed
Implementing HDFSWriter to Kafka consumer record processing.
Tiihott
committed
Implementing HDFSWriter to Kafka consumer record processing (2).
Tiihott
committed
Implementing HDFSWriter to Kafka consumer record processing (3).
Tiihott
committed
Testing configurations, setting up testcases, debugging dependency issues.
Tiihott
committed
Testing and debugging kafka consumer processing, debugging dependency issues.
Tiihott
committed
Fixed issue in Kafka mock consumer processing, continued testing Avro-serialization in Kafka consumer processing.
Tiihott
committed
Added Kerberos authentication to HDFS access.
Tiihott
committed
Added configurations for Kerberos authentication to HDFS access.
Tiihott
committed
Added support to define mock data amount. Debugging AVRO-serialization of records to specific sized files.
Tiihott
committed
Debugging issue on how AVRO-file size is tracked.
Tiihott
committed
Changed AVRO-serialization to use approximated file size values instead of definite values as definite requires continuous use of flush() after appending records to AVRO.
Tiihott
committed
Still having issues in the filesize of AVRO-serialization.
Tiihott
committed
Rolled back to using definitive file size values on avro-serialization files with flush() after every record append. Improved test cases.
Tiihott
committed
Setting up HDFS access tests.
Tiihott
committed
Setting up HDFS access tests (2).
Tiihott
committed
Added readme and issue template.
Tiihott
committed
Fixed faulty origin and source parameter generation in DatabaseOutput.java.
Tiihott
committed
Replacing mxj_01 based performance metrics with dropwizard metrics implementation. (WIP)
Tiihott
committed
Implemented Dropwizard metrics to monitor Kafka consumer processing performance. Cleaned up namings in in DurationStatistics. Debugging faulty consumed kafka topic partition ordering.
Tiihott
committed
Debugging faulty kafka topic partition ordering. (2)
Tiihott
committed
Changed logic on how topic partitions are handled, each partition now gets its own exclusive AVRO-file queueNamePrefix which is used to generate the AVRO-files where the records of the partition ar…
Tiihott
committed
Altered Kafka MockConsumer to use subscription instead of assign method.
Tiihott
committed
Implementing fix to consumer group creation for partitions.
Tiihott
committed
Added config parameter for defining the number of consumers in a consumer group.
Tiihott
committed
Fixed consumer group generation and its tests. Added example configuration files to test folder.
Tiihott
committed
Cleaned up and improved tests.
Tiihott
committed
Improved commenting. Cleaned up code. Cleaned up class names.
Tiihott
committed

Commits on Jan 30, 2024

Commits on Feb 1, 2024

Setting up tests for HDFS write.
Tiihott
committed

Commits on Feb 5, 2024

Setting up MiniCluster for HDFS testing. Debugging issues in HDFS tests.
Tiihott
committed

Commits on Feb 6, 2024

Commits on Feb 7, 2024

Cleaning up comments. Adding missing loggers to files.
Tiihott
committed

Commits on Feb 8, 2024

Commits on Feb 9, 2024

Testing FileSystem limitations on how pruning of the HDFS database can be achieved. (2)
Tiihott
committed

Commits on Feb 12, 2024

Setting up pruning using MapReduce.
Tiihott
committed

Commits on Feb 13, 2024

Setting up pruning using avro-mapred.
Tiihott
committed

Commits on Feb 15, 2024

Commits on Feb 16, 2024

Attempt at optimizing the output of MapReduce.
Tiihott
committed

Commits on Feb 19, 2024

Reverting back to using FileSystem modification timestamps for pruning instead of MapReduce. Implemented HDFSPrune.java which prunes the HDFS database.
Tiihott
committed

Commits on Feb 20, 2024

Parametrized the pruning cutoff epoch offset. Removed obsolete parameters from Config.java. Fixed CombinedFullTest.java to mirror the changes made to the code. Fixed AVRO maximumFileSize parametriz…
Tiihott
committed

Commits on Feb 21, 2024

Added the missing kerberized access to HDFSPrune.java.
Tiihott
committed

Commits on Feb 23, 2024

Implementing fixes/improvements from code review. Fixes for comments 1, 2, 3, 4, 5, 6, 7, 9, 11, 12 and 13.
Tiihott
committed

Commits on Feb 26, 2024

Implemented null object pattern for RecordOffsetObject (improvements from code review comments 14 and 15).
Tiihott
committed

Commits on Mar 13, 2024

Reverted back to using original modification timestamp functionality of the FileSystem in HDFS and file pruning because Kafka records are not guaranteed to be ordered by time.
Tiihott
committed

Commits on Apr 4, 2024

Changed Offset class namings. Changed file system asset naming.
Tiihott
committed

Commits on May 15, 2024

Fixed bug in topic partition offset map creation in HDFSRead.java. Added another mock hdfs file for testing.
Tiihott
committed

Commits on May 17, 2024

Commits on May 27, 2024

Commits on May 28, 2024

Commits on May 30, 2024

Commits on May 31, 2024

Commits on Jun 3, 2024

Commits on Jun 4, 2024

Deleted QueueUtilities.java and moved the contained methods to WritableQueue.java
Tiihott
committed

Commits on Jun 11, 2024

Added ConfigTest.java.
Tiihott
committed

Commits on Jun 12, 2024

Commits on Jun 13, 2024

Commits on Jun 17, 2024

Improved test assertion setups.
Tiihott
committed

Commits on Jun 20, 2024

Commits on Jun 24, 2024

Commits on Jun 27, 2024

Commits on Jul 2, 2024

Commits on Jul 3, 2024

Added more configuration checks. Changed default directory of application.properties. Updated example configuration file.
Tiihott
committed

Commits on Jul 4, 2024

Commits on Aug 8, 2024

Fix for issue #32 by enabling automatic TGT renewal.

Tiihott
authored and
StrongestNumber9
committed
Replaced hadoop Configuration class usage with the child class HdfsConfiguration.

Tiihott
authored and
StrongestNumber9
committed

Commits on Aug 20, 2024